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Preface 


View the promotional video on YouTube 


These are my lecture notes for my online Coursera course, Matrix Algebra for Engineers. I have 
divided these notes into chapters called Lectures, with each Lecture corresponding to a video on 
Coursera. I have also uploaded all my Coursera videos to YouTube, and links are placed at the top of 
each Lecture. 

There are problems at the end of each lecture chapter and I have tried to choose problems that 
exemplify the main idea of the lecture. Students taking a formal university course in matrix or linear 
algebra will usually be assigned many more additional problems, but here I follow the philosophy 
that less is more. I give enough problems for students to solidify their understanding of the material, 
but not too many problems that students feel overwhelmed and drop out. I do encourage students to 
attempt the given problems, but if they get stuck, full solutions can be found in the Appendix. 

There are also additional problems at the end of coherent sections that are given as practice quizzes 
on the Coursera platform. Again, students should attempt these quizzes on the platform, but if a 
student has trouble obtaining a correct answer, full solutions are also found in the Appendix. 

The mathematics in this matrix algebra course is at the level of an advanced high school student, but 
typically students would take this course after completing a university-level single variable calculus 
course. There are no derivatives and integrals in this course, but student’s are expected to have a 
certain level of mathematical maturity. Nevertheless, anyone who wants to learn the basics of matrix 


algebra is welcome to join. 


JEFFREY R. CHASNOV 
Hong Kong 
July 2018 
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Week I 


Matrices 


In this week’s lectures, we learn about matrices. Matrices are rectangular arrays of numbers or 
other mathematical objects and are fundamental to engineering mathematics. We will define matrices 
and how to add and multiply them, discuss some special matrices such as the identity and zero matrix, 


learn about transposes and inverses, and define orthogonal and permutation matrices. 


Lecture 1 


Definition of a matrix 


View this lecture on YouTube 


An m-by-n matrix is a rectangular array of numbers (or other mathematical objects) with m rows 


and n columns. For example, a two-by-two matrix A, with two rows and two columns, looks like 


ea) 


The first row has elements a and b, the second row has elements c and d. The first column has elements 
a and c; the second column has elements b and d. As further examples, two-by-three and three-by-two 


b aod 

a Cc 

-(; ). C=1]b e 
e f ay 


Of special importance are column matrices and row matrices. These matrices are also called vectors. 


matrices look like 


The column vector is in general n-by-one and the row vector is one-by-n. For example, when n = 3, 


we would write a column vector as 


and a row vector as 
y= (a b c) ‘ 


A useful notation for writing a general m-by-n matrix A is 


44, 442 Mn 

421 422 42n 
A = 

Am1 @m2 °*** Amn 


Here, the matrix element of A in the ith row and the jth column is denoted as aij. 


6 LECTURE 1. DEFINITION OF A MATRIX 


Problems for Lecture 1 


1. The diagonal of a matrix A are the entries a;; where i = j. 
a) Write down the three-by-three matrix with ones on the diagonal and zeros elsewhere. 
b) Write down the three-by-four matrix with ones on the diagonal and zeros elsewhere. 
c) Write down the four-by-three matrix with ones on the diagonal and zeros elsewhere. 


Solutions to the Problems 


Lecture 2 


Addition and multiplication of 


matrices 


View this lecture on YouTube 


Matrices can be added only if they have the same dimension. Addition proceeds element by element. 


eB) fe of) 2 (ere Uses 
c d gh} \ct+g dth)’ 


Matrices can also be multiplied by a scalar. The rule is to just multiply every element of the matrix. 


~{% 5\_[ # 
c dad) \kc kd)° 


Matrices (other than the scalar) can be multiplied only if the number of columns of the left matrix 


For example, 
For example, 


equals the number of rows of the right matrix. In other words, an m-by-n matrix on the left can only 
be multiplied by an n-by-k matrix on the right. The resulting matrix will be m-by-k. Evidently, matrix 
multiplication is generally not commutative. We illustrate multiplication using two 2-by-2 matrices: 


a b\ fe f\  fae+bg af+bh e f\ fa b\ faetcf be+df 
c d)\g h} \ce+dg cf+dh}’ g h}) \c dd) \ag+ch bg+dh}’ 


First, the first row of the left matrix is multiplied against and summed with the first column of the right 


matrix to obtain the element in the first row and first column of the product matrix. Second, the first 
row is multiplied against and summed with the second column. Third, the second row is multiplied 
against and summed with the first column. And fourth, the second row is multiplied against and 
summed with the second column. 

In general, an element in the resulting product matrix, say in row i and column j, is obtained by 
multiplying and summing the elements in row i of the left matrix with the elements in column j of 
the right matrix. We can formally write matrix multiplication in terms of the matrix elements. Let A 
be an m-by-n matrix with matrix elements a;; and let B be an n-by-p matrix with matrix elements Jjj. 


Then C = AB is an m-by-p matrix, and its ij matrix element can be written as 
n 
cy =) aixbyy. 
k=1 


Notice that the second index of a and the first index of b are summed over. 


8 LECTURE 2. ADDITION AND MULTIPLICATION OF MATRICES 


Problems for Lecture 2 


1. Define the matrices 


Compute if defined: B-—2A, 3C—E, AC, CD, CB. 


2. Let A= ne ,B= ee and C = aes . Verify that AB = AC and yet B £ C. 
2 4 13 0 2 


111 2 0 0 
3. LetA=]1 2 3] andD=1]0 3 OJ]. Compute AD and DA. 
13 4 00 4 


4. Prove the associative law for matrix multiplication. That is, let A be an m-by-n matrix, B an n-by-p 
matrix, and C a p-by-q matrix. Then prove that A(BC) = (AB)C. 


Solutions to the Problems 


Lecture 3 


Special matrices 


View this lecture on YouTube 


The zero matrix, denoted by 0, can be any size and is a matrix consisting of all zero elements. Multi- 
plication by a zero matrix results in a zero matrix. The identity matrix, denoted by I, is a square matrix 
(number of rows equals number of columns) with ones down the main diagonal. If A and I are the 
same sized square matrices, then 

AI=IJA=A, 


and multiplication by the identity matrix leaves the matrix unchanged. The zero and identity matrices 
play the role of the numbers zero and one in matrix multiplication. For example, the two-by-two zero 


and identity matrices are given by 


A diagonal matrix has its only nonzero elements on the diagonal. For example, a two-by-two diagonal 


pate ol: 
0 do 


Usually, diagonal matrices refer to square matrices, but they can also be rectangular. 


matrix is given by 


A band (or banded) matrix has nonzero elements only on diagonal bands. For example, a three-by- 
three band matrix with nonzero diagonals one above and one below a nonzero main diagonal (called 


a tridiagonal matrix) is given by 


dy ay 0 
B= by do a2 
0 b ds 


An upper or lower triangular matrix is a square matrix that has zero elements below or above the 


diagonal. For example, three-by-three upper and lower triangular matrices are given by 
0 
0 


a 0 
, L=]bd 
Ge eo Af 


a b 
U=10 d 
0 0 f 


10 LECTURE 3. SPECIAL MATRICES 


Problems for Lecture 3 


1. Let 
—1 2 
A= : 


Construct a two-by-two matrix B such that AB is the zero matrix. Use two different nonzero columns 


for B. 
ay 0 by 0 = ay by 0 
0 am/\0 b&b) \ 0 abo] 


Prove in general that the product of two diagonal matrices is a diagonal matrix, with elements given 


2. Verify that 


by the product of the diagonal elements. 


a az) (by by\ _ [arb aybz + agbs 
0 a3) \0 bg 0 a3b3 


Prove in general that the product of two upper triangular matrices is an upper triangular matrix, with 


3. Verify that 


the diagonal elements of the product given by the product of the diagonal elements. 


Solutions to the Problems 


Practice quiz: Matrix definitions 


1. Identify the two-by-two matrix with matrix elements a;; = i — j. 


1 -1 —1 1 
2. The matrix product ( ( ; is equal to 


—2 2 
°C 4) 


d) —2 -2 
2 2 
n 
3. Let A and B be n-by-n matrices with (AB); = x ajzby;. If A and B are upper triangular matrices, 
k=1 

then a;, = 0 or Dx; = 0 when 
A.k <i Bk>i Ck<j D.k>j 

a) A and C only 

b) A and D only 

c) Band C only 


d) Band D only 


Solutions to the Practice quiz 
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LECTURE 3. SPECIAL MATRICES 


Lecture 4 


Transpose matrix 


View this lecture on YouTube 


The transpose of a matrix A, denoted by AT and spoken as A-transpose, switches the rows and 


columns of A. That is, 


441 42 *** Ain 441 421 +** Ami 
. 421 422 °+*: A2n T 412 422 ++: aAm2 
ifA= , then At = 

Ami 4m2 °** Amn Qn G2 *** Amn 


The following are useful and easy to prove facts: 
T zt T T T 
(A ) =A, and (A+B)’ =aT +B", 


A less obvious fact is that the transpose of the product of matrices is equal to the product of the 


transposes with the order of multiplication reversed, i.e., 
(AB)? = BTA’, 


If A is a square matrix, and A! = A, then we say that A is symmetric. If AT = —A, then we say that A 
is skew symmetric. For example, three-by-three symmetric and skew symmetric matrices look like 


b c 0 b c 
d e|, —b e 
cn ar | —c -e 0 


Notice that the diagonal elements of a skew-symmetric matrix must be zero. 


13 


14 LECTURE 4. TRANSPOSE MATRIX 


Problems for Lecture 4 


1. Prove that (AB)' = BTAT. 


2. Show using the transpose operator that any square matrix A can be written as the sum of a sym- 


metric and a skew-symmetric matrix. 
3. Prove that ATA is symmetric. 


Solutions to the Problems 


Lecture 5 


Inner and outer products 


View this lecture on YouTube 


The inner product (or dot product or scalar product) between two vectors is obtained from the ma- 
trix product of a row vector times a column vector. A row vector can be obtained from a column 
vector by the transpose operator. With the 3-by-1 column vectors u and v, their inner product is given 
by 
O71 
uly = (1 uz u3) 02 | = 4101 + U202 + 303. 
03 


If the inner product between two nonzero vectors is zero, we say that the vectors are orthogonal. The 


norm of a vector is defined by 
1/2 1/2 
\[uj| = (u"w) = (uj ++ 18) . 


If the norm of a vector is equal to one, we say that the vector is normalized. If a set of vectors are 
mutually orthogonal and normalized, we say that these vectors are orthonormal. 
An outer product is also defined, and is used in some applications. The outer product between u 


and v is given by 


uy uy, =%Uuyv2 Uy,03 
uv! = | u2 (a 02 23) = | Ug, U2V2 U2V3 
U3 U30, U302 U303 


Notice that every column is a multiple of the single vector u, and every row is a multiple of the single 


vector v!. 


15 


16 LECTURE 5. INNER AND OUTER PRODUCTS 


Problems for Lecture 5 


ad 
1. Let A be a rectangular matrix givenby A= | be | . Compute ATA and show that itis a symmetric 


c f 


square matrix and that the sum of its diagonal elements is the sum of the squares of all the elements 
of A. 


2. The trace of a square matrix B, denoted as TrB, is the sum of the diagonal elements of B. Prove that 
Tr(ATA) is the sum of the squares of all the elements of A. 


Solutions to the Problems 


Lecture 6 


Inverse matrix 


View this lecture on YouTube 


Square matrices may have inverses. When a matrix A has an inverse, we say it is invertible and 


denote its inverse by A7!. The inverse matrix satisfies 
AAT=ATA=L 
If A and B are invertible matrices, then (AB)! = B-!A7!. Furthermore, if A is invertible then so is 


At and (At) = (A?) 
It is illuminating to derive the inverse of a general 2-by-2 matrix. Write 


CDE ZC) 


and try to solve for x1, y1, X2 and yz in terms of a, b, c, and d. There are two inhomogeneous and two 


homogeneous linear equations: 


ax, + by, = 1, cx1 + dy; =0, 


cx2 + dy2 = 1, axa + by2 = 0. 


To solve, we can eliminate y; and y2 using the two homogeneous equations, and find x; and x2 using 


the two inhomogeneous equations. The solution for the inverse matrix is found to be 


The term ad — bc is just the definition of the determinant of the two-by-two matrix: 


det (' ;) =a 
cfd 


The determinant of a two-by-two matrix is the product of the diagonals minus the product of the 
off-diagonals. Evidently, a two-by-two matrix A is invertible only if det A 4 0. Notice that the inverse 
of a two-by-two matrix, in words, is found by switching the diagonal elements of the matrix, negating 
the off-diagonal elements, and dividing by the determinant. 

Later, we will show that an n-by-n matrix is invertible if and only if its determinant is nonzero. 


This will require a more general definition of the determinant. 


17 


18 LECTURE 6. INVERSE MATRIX 


Problems for Lecture 6 


1. Find the inverses of the matrices 6 and ete : 
4 5 3 3 

2. Prove that if A and B are same-sized invertible matrices , then (AB)~! = B-'A7!. 

3. Prove that if A is invertible then so is AT, and (AT)~! = (A~!)?. 


4. Prove that if a matrix is invertible, then its inverse is unique. 


Solutions to the Problems 


Practice quiz: Transpose and inverses 
1. (ABC)! is equal to 

a ABC 

b) A'C™BI 

c) CTATBI 

d) CIBTAT 
2. Suppose A is a square matrix. Which matrix is not symmetric? 

a) A+ AT 

b) AAT 

c) A—Al 


d) ATA 


3. Which matrix is the inverse of é ) ? 
a 1 2 —2 
2 =] 2 
b) 
A 1 2 2 
2\-1 -2 
1/-2 -—2 
d) = 


Solutions to the Practice quiz 
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LECTURE 6. INVERSE MATRIX 


Lecture 7 


Orthogonal matrices 


View this lecture on YouTube 


A square matrix Q with real entries that satisfies 
Oo = Qt 
is called an orthogonal matrix. Another way to write this definition is 
QQ! =I and QTQ=1. 


We can more easily understand orthogonal matrices by examining a general two-by-two example. Let 
Q be the orthogonal matrix given by 


a= (MR) =(@ 9), 


where q and qp are the two-by-one column vectors of the matrix Q. Then 


T T T 
q Wd 9192 
q3 ( ) Bq 342 
If Q is orthogonal, then Q'OQ =I and 


gig = =1 and qiqe=q7q1 =0. 


That is, the columns of Q form an orthonormal set of vectors. The same argument can also be made 
for the rows of Q. 

Therefore, an equivalent definition of an orthogonal matrix is a square matrix with real entries 
whose columns (and also rows) form a set of orthonormal vectors. 

There is a third equivalent definition of an orthogonal matrix. Let Q be an n-by-n orthogonal 
matrix, and let x be an n-by-one column vector. Then the length squared of the vector Ox is given by 


[[Ox|/? = (x) (Ox): =O" Ox = sk = xe = (LIF, 


The length of Ox is therefore equal to the length of x, and we say that an orthogonal matrix is a matrix 
that preserves lengths. In the next lecture, an example of an orthogonal matrix will be the matrix that 
rotates a two-dimensional vector in the plane. 
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22 LECTURE 7. ORTHOGONAL MATRICES 


Problems for Lecture 7 
1. Show that the product of two orthogonal matrices is orthogonal. 
2. Show that the n-by-n identity matrix is orthogonal. 


Solutions to the Problems 


Lecture 8 


Rotation matrices 


View this lecture on YouTube 


A matrix that rotates a vector in space doesn’t change the vector’s length and so should be an orthog- 


Rotating a vector in the x-y plane. 


onal matrix. Consider the two-by-two rotation matrix that rotates a vector through an angle @ in the 
x-y plane, shown above. Trigonometry and the addition formula for cosine and sine results in 


x’ =rcos(6+ yp) y =rsin(@+y) 
= r(cos @ cos p — sin@ sin i) = r(sin@ cos + cos @ sin p) 
= xcos@—ysin@ = xsin@+ycosé. 


Writing the equations for x’ and y’ in matrix form, we have 


x'\  [cos@ —siné\ (x 
y}  \sin@ cos} \y)’ 


The above two-by-two matrix is a rotation matrix and we will denote it by Rg. Observe that the rows 
and columns of Rg are orthonormal and that the inverse of Rg is just its transpose. The inverse of Rg 


rotates a vector by —6. 


23 


24 LECTURE 8. ROTATION MATRICES 


Problems for Lecture 8 


cos@ —sin@ 


1. Let R(@) = ( ) Show that R(—@) = R(@)71. 


sin 0 cos 8 


2. Find the three-by-three matrix that rotates a three-dimensional vector an angle @ counterclockwise 
around the z-axis. 


Solutions to the Problems 


Lecture 9 


Permutation matrices 


View this lecture on YouTube 


Another type of orthogonal matrix is a permutation matrix. A permutation matrix, when multiplying 
on the left, permutes the rows of a matrix, and when multiplying on the right, permutes the columns. 
Clearly, permuting the rows of a column vector will not change its length. 

For example, let the string {1,2} represent the order of the rows of a two-by-two matrix. Then 
the two possible permutations of the rows are given by {1,2} and {2,1}. The first permutation is 
no permutation at all, and the corresponding permutation matrix is simply the identity matrix. The 


second permutation of the rows is achieved by 


pos 


The rows of a three-by-three matrix have 3! = 6 possible permutations, namely {1,2,3}, {1,3,2}, 
{2,1,3}, {2,3,1}, {3,1,2}, {3,2,1}. For example, the row permutation {3,1,2} is achieved by 


bc g hi 
e fl=la be 
h de f 


Notice that the permutation matrix is obtained by permuting the corresponding rows of the identity 


0 0 1 
1 0 O 
0 1 0 


oe NA 


i 


matrix, with the rows of the identity matrix permuted as {1,2,3} — {3,1,2}. That a permutation 
matrix is just a row-permuted identity matix is made evident by writing 


PA = (PI)A, 


where P is a permutation matrix and PI is the identity matrix with permuted rows. The identity matrix 
is orthogonal, and so is the permutation matrix obtained by permuting the rows of the identity matrix. 
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26 LECTURE 9. PERMUTATION MATRICES 


Problems for Lecture 9 


1. Write down the six three-by-three permutation matrices corresponding to the permutations {1,2,3}, 
{1,3,2}, {2,1,3}, {2,3,1}, {3,1,2}, {3,2, 1}. 


2. Find the inverses of all the three-by-three permutation matrices. Explain why some matrices are 
their own inverses, and others are not. 


Solutions to the Problems 


Practice quiz: Orthogonal matrices 


1. Which matrix is not orthogonal? 


(io) 


1 0 
»(, 4) 
01 
at 
i i 
°( 7) 


2. Which matrix rotates a three-by-one column vector an angle @ counterclockwise around the x-axis? 


1 0 0 
0 cos@? —siné 


a) 


f sin @ cos 6 
sind 0 cos 0 
b) 0 1 0 
cos@ 0 —sinéd 
cos@ —sin@ 0 
sin 0 cos@ 0 

| } 


c) 
0 0 1 


cos@ sin@ 
—sin@ cos@ 
0 0 1 


d) 


27 


28 LECTURE 9. PERMUTATION MATRICES 


3. Which matrix, when left multiplying another matrix, moves row one to row two, row two to row 


three, and row three to row one? 


0 1 0 
a)1}0 01 
1 0 0 


001 
b) 


oOo Ff 
- oO 
oo 


c) |}0 1 0 


= 
oO 
(as) 


mR 
oO 
(an) 


A) Goe 4 


oO 
a 
i=) 


Solutions to the Practice quiz 


Week II 


Systems of Linear Equations 


31 


In this week’s lectures, we learn about solving a system of linear equations. A system of linear 
equations can be written in matrix form, and we can solve using Gaussian elimination. We will learn 
how to bring a matrix to reduced row echelon form, and how this can be used to compute a matrix 
inverse. We will also learn how to find the LU decomposition of a matrix, and how to use this 
decomposition to efficiently solve a system of linear equations. 


32 


Lecture 10 


Gaussian elimination 


View this lecture on YouTube 


Consider the linear system of equations given by 


3x1 + 2x2 — x3 = —1, 


6X1 = 6X2 == 7X3 = —7, 


3x1 — 4x2 + 4x3 = —6, 


which can be written in matrix form as 


8 2 lh 7a -1 
6 —6 7 xy] = —7\, 
3 4 A) \xs 6 


or symbolically as Ax = b. 


The standard numerical algorithm used to solve a system of linear equations is called Gaussian 
elimination. We first form what is called an augmented matrix by combining the matrix A with the 


column vector b: 
—3 2 -1 -1 
6 —6 7 —7 


Row reduction is then performed on this augmented matrix. Allowed operations are (1) interchange 
the order of any rows, (2) multiply any row by a constant, (3) add a multiple of one row to another 
row. These three operations do not change the solution of the original equations. The goal here is 
to convert the matrix A into upper-triangular form, and then use this form to quickly solve for the 


unknowns x. 


We start with the first row of the matrix and work our way down as follows. First we multiply the 
first row by 2 and add it to the second row. Then we add the first row to the third row, to obtain 


—-3 2 -1 -1 


34 LECTURE 10. GAUSSIAN ELIMINATION 
We then go to the second row. We multiply this row by —1 and add it to the third row to obtain 
—3 2 -1 -1 


Oh o=2- - SbY = 9 


The original matrix A has been converted to an upper triangular matrix, and the transformed equations 


can be determined from the augmented matrix as 


3x1 + 2x2 — x3 = —-1, 
—2x2 + 5x3 = —9, 
—2x3 = 2; 


These equations can be solved by back substitution, starting from the last equation and working 
backwards. We have 


x3 = -1, 
1 
x2= —=(—9 — 5x3) ee 
2 
x= = 5(-1+ x5 — 2x2) =12, 


We have thus found the solution 


x41 2 
x2), = 2 
x3, —1 


When performing Gaussian elimination, the matrix element that is used during the elimination proce- 
dure is called the pivot. To obtain the correct multiple, one uses the pivot as the divisor to the matrix 
elements below the pivot. Gaussian elimination in the way done here will fail if the pivot is zero. If 
the pivot is zero, a row interchange must first be performed. 

Even if no pivots are identically zero, small values can still result in an unstable numerical compu- 
tation. For very large matrices solved by a computer, the solution vector will be inaccurate unless row 
interchanges are made. The resulting numerical technique is called Gaussian elimination with partial 


pivoting, and is usually taught in a standard numerical analysis course. 


35 


Problems for Lecture 10 


1. Using Gaussian elimination with back substitution, solve the following two systems of equations: 


(a) 
3X1 = 7X2 = 2x3 = —7, 
—3x4 + 5X2 +xX3= 5, 
6x, — 4x2 = 2. 
(b) 


xy — 2x2 +3x3 = 1, 


X14 + 3x2 —-2%3 = —1, 


2x1 — 5x2 + 5x3 = 1. 


Solutions to the Problems 
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LECTURE 10. GAUSSIAN ELIMINATION 


Lecture 11 


Reduced row echelon form 


View this lecture on YouTube 


A matrix is said to be in reduced row echelon form if the first nonzero entry in every row is a one, all 

the entries below and above this one are zero, and any zero rows occur at the bottom of the matrix. 
The row elimination procedure of Gaussian elimination can be continued to bring a matrix to 

reduced row echelon form. We notate the reduced row echelon form of a matrix A as rref(A). For 


example, consider the three-by-four matrix 
123 4 
A=1/14 5 6 7 
6 7 8 9 
Row elimination can proceed as 


123 4 1 2 3 4 123 4 
45 6 7 > 10 3 6 9 >10 12 3)47)0 1 2 3]; 
6 7 8 9 0 -—5 -—10 —15 01 2 3 


and we therefore have 


1 0 -1 -2 
rref(A)=1]0 1 2 3 
0 O O 0 


We say that the matrix A has two pivot columns, that is, two columns that contain a pivot position 
with a one in the reduced row echelon form. 

Note that rows may need to be exchanged when computing the reduced row echelon form. Also, 
the reduced row echelon form of a matrix A is unique, and if A is a square invertible matrix, then 


rref(A) is the identity matrix. 
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38 LECTURE 11. REDUCED ROW ECHELON FORM 


Problems for Lecture 11 


1. Put the following matrices into reduced row echelon form and state which columns are pivot 


columns: 
(a) 
3 -7 -—2 -7 
A=]|-3 5 1 5 
6 —4 0 2 
(b) 
12 1 
A= [2 4.1 
3 6 2 


Solutions to the Problems 


Lecture 12 


Computing inverses 


View this lecture on YouTube 


By bringing an invertible matrix to reduced row echelon form, that is, to the identity matrix, we 


can compute the matrix inverse. Given a matrix A, consider the equation 
AAS eT, 


for the unknown inverse A~!. Let the columns of A~! be given by the vectors a; L ay te and so on. 
The matrix A multiplying the first column of A7! is the equation 


Aa;!=e ith = ( 7 
1, =e, withe;=(1 0... 0), 


and where e, is the first column of the identity matrix. In general, 


-1 
Aa, = &, 

for i = 1,2,...,n. The method then is to do row reduction on an augmented matrix which attaches 

the identity matrix to A. To find A—!, elimination is continued until one obtains rref(A) =I. 


We illustrate below: 


3 2 -1 1 0 0 —3 2 -1 1 0 0 
6 —6 7 0 1 ol—- 0 -2 5 2 1 0oj|—- 
3-4 4 0 0 1 0 -2 3 1 0 1 
—3 2 —-1 1 0 0 —3 0 4 3 1 0 
0 -—2 5 2 1 Oo|—- 0 -2 5 2 1 0ojJ—- 
0 0 -—-2 -1 -I1 1 0 0 —2 -1 -1 1 
3 0 0 1 —1 2 1 0 0 —-1/3 1/3. —2/3 
0 -2 0 —-1/2 —3/2 5/2) > 0 1 0 1/4 3/4 —5/4]|; 
0 0 —2 -!1 —1 1 0 0 1 1/2 1/2 —-1/2 
and one can check that 
—3 2 —-1 -1/3 1/3. —2/3 100 
6 —6 7 1/4 3/4 —5/4|=1]0 1 0 
3-4 4 1/2 1/2 —-1/2 001 
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Problems for Lecture 12 


1. Compute the inverse of 


Solutions to the Problems 


LECTURE 12. COMPUTING INVERSES 


Practice quiz: Gaussian elimination 


1. Perform Gaussian elimination without row interchange on the following augmented matrix: 
1 -—2 1 0 
2 1 -3 5 |. Which matrix can be the result? 
4 —7 1 -2 


a) 


i) 
i) 
| 
N 
| 
ies) 


_ 
| 
N 
— 
j=) 


b) 1 -1 1 
0 0 -2 3 
c) 1 -1 1 
0 0 -3 —-2 
1 -2 1 0 
d) |0 1 -1 1 


| 
| 
ee 
| 


2. Which matrix is not in reduced row echelon form? 


100 2 
0 3 
0 1 2 


a) 


— 
N 
oO 


0 


( 
| 


oO 
i=) 
oO 
a 


c) 


d) 


fj) 
or 
FP OF 
oS Rr Oo°O 
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3-7 -2 
3. The inverse of | —3 5 1] is 
6 —4 0 
4/3 2/3 1/2 
a) 2 1 1/2 
-3 -5 —-1 
2/3 1/2 4/3 
b) 1 1/2 2 
—3 -5 -1 
2/3 4/3 1/2 
c) 1 2 1/2 
-5 -3 —-1 
2/3 4/3 1/2 
d) 1 2 1/2 
—-3 -5 —-1 


Solutions to the Practice quiz 


LECTURE 12. COMPUTING INVERSES 


Lecture 13 


Elementary matrices 


View this lecture on YouTube 


The row reduction algorithm of Gaussian elimination can be implemented by multiplying elemen- 
tary matrices. Here, we show how to construct these elementary matrices, which differ from the 
identity matrix by a single elementary row operation. Consider the first row reduction step for the 


following matrix A: 


—3 2 -1 -—3 2 -1 1 0 0 
A= 6 —6 7 | 0 —2 5] =M,A, whereM,;=]2 1 O 
3 -4 4 3 -4 4 001 


To construct the elementary matrix M,, the number two is placed in column-one, row-two. This matrix 
multiplies the first row by two and adds the result to the second row. 


The next step in row elimination is 


—3 2 —-1 —3 2 —-1 100 
0 -—2 5|—-> 0 -2 5 | =MoM,A, whereMo=]0 1 0O 
3-4 4 0 -—2 3 1041 


Here, to construct My the number one is placed in column-one, row-three, and the matrix multiplies 
the first row by one and adds the result to the third row. 


The last step in row elimination is 


—3 2 -1 —3 2 -1 1 0 0 
0 -—2 5|—- Q -2 5 | =M3MoM,A,_ where M3 = 0 1 0 
0 -2 3 0 0 -2 0 -1 1 


Here, to construct M3 the number negative-one is placed in column-two, row-three, and this matrix 
multiplies the second row by negative-one and adds the result to the third row. 
We have thus found that 
M3M2M),A = U, 


where U is an upper triangular matrix. This discussion will be continued in the next lecture. 
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44 LECTURE 13. ELEMENTARY MATRICES 


Problems for Lecture 13 


1. Construct the elementary matrix that multiplies the second row of a four-by-four matrix by two and 
adds the result to the fourth row. 


Solutions to the Problems 


Lecture 14 


LU decomposition 


View this lecture on YouTube 


In the last lecture, we have found that row reduction of a matrix A can be written as 
M3M2M),A = U, 
where U is upper triangular. Upon inverting the elementary matrices, we have 
A =M,'M;'M;'U. 


Now, the matrix M, multiples the first row by two and adds it to the second row. To invert this 
operation, we simply need to multiply the first row by negative-two and add it to the second row, so 
that 


1 0 0 1 6 
M=|2 2.0), Mo =)|-2 2 
Ot 4 0 0 
Similarly, 
1 0 0 10 0 1 0 1 0 0 
Me=|0 1.0/5 Mi’ =| G1 O]s Me=)0 1 OO] M=l0 1 0 
Dg es (ae 6 0 =] O° a 
Therefore, 
L=M,"M,"M,* 
is given by 
1 0 0 1 0 0\ fi 0 0 1 0 
ba} 2 1 01 0/{0 1 oJ =|-2 1 J 
0.0 1 \=1 @ 1/7 AD a 1 1 1 1 


which is lower triangular. Also, the non-diagonal elements of the elementary inverse matrices are 


simply combined to form L. Our LU decomposition of A is therefore 


=3. 2 <1 1 0 0 -3 2 -1 
6 -6 7|=]-2 1 0 -2 5 
3 -4 4 -1 11 0 0 —2 
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Problems for Lecture 14 


1. Find the LU decomposition of 


Solutions to the Problems 


LECTURE 14. LU DECOMPOSITION 


Lecture 15 


Solving (LU)x = b 


View this lecture on YouTube 


The LU decomposition is useful when one needs to solve Ax = b for many right-hand-sides. With the 


LU decomposition in hand, one writes 
(LU)x = L(Ux) =b, 


and lets y = Ux. Then we solve Ly = b for y by forward substitution, and Ux = y for x by backward 
substitution. It is possible to show that for large matrices, solving (LU)x = b is substantially faster 


than solving Ax = b directly. 


We now illustrate the solution of LUx = b, with 


1 0 —3 2 -1 —1 
L=]-2 1 , U= 0 -—2 S51], b=]|-7 
-1 1 0 0 —2 —6 


1 0 0 Y1 —1 
—2 1 0 yo | = | -7 
-1 141 Y3 —6 
Using forward substitution 
y 1 = dy 


We then solve Ux = y, that is 


3 2 -1\ [x -1 
0 -2 5 x2} = | -9 
0 0 -—2 x3, 2 
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Using back substitution, 


LECTURE 15. SOLVING (LU)X = B 


x3 = -1, 
1 

x2 = —5(-9— 54g) = 
1 

=— 1— 2x4 
xy 3( oo) 
and we have found 

x1 2 
x2) = 2 
x3 —1 


Problems for Lecture 15 


1. Using 
3 -7 -2 
A=]|-3 5 1 
6 —4 0 


compute the solution to Ax = b with 


-3 1 
(ajb=] 3], (b)b=]-1 
2 1 


Solutions to the Problems 
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LECTURE 15. SOLVING (LU)X = B 


Practice quiz: LU decomposition 


1. Which of the following is the elementary matrix that multiplies the second row of a four-by-four 
matrix by 2 and adds the result to the third row? 


Co ON 
oor oO 
oreo 
> ee >) 


b) 


| 


oo lO Fe 
oor o& 
oOrRFN OO 


oo oO Fe 
OoNF OO 
oreo 


d) 


Nu OO 
oo Ff Oo 
or oO CO 
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52 LECTURE 15. SOLVING (LU)X = B 


3 -7 -2 
2. Which of the following is the LU decomposition of |-—3 5 1/4? 
6 -4 0 
1 0 0 3. =7 2 
a)|}-1 1 0 Q —2 —1 
2 -5 1/2) \O0 O -2 
1 OO O\ /3 -7 -2 
b)}-1 1 0 0 -—2 -1 
2-5 1 0 oO -1 
1 0 0\ /3 -7 -2 
c) | -1 2 1 0 -1 -1 
2 -10 6/ \O O -1 
1 oO 0 3. =F <2 
d)|-1 1 O 0 -2 -1 
4 -5 1 -6 14 3 
1 oO 0 3 =7 =2 1 
3. SupposeL=]}-1 1 O}],U=1]0 -—2 -1],andb= | -1 
2-5 1 0 oO -1 1 
y = Ux. The solutions for y and x are 
=1 1/6 
ayy= O|,x=]1/2 
1 —1 
1 —1/6 
b) y= O},x=]-1/2 
-1 1 
1 1/6 
cy= O|,x= ]| —-1/2 
—1 1 
=1 —1/6 
d)y= 0|,x= 1/2 
1 1 


Solutions to the Practice quiz 


. Solve LUx = b by letting 


Week III 


Vector Spaces 


55 


In this week’s lectures, we learn about vector spaces. A vector space consists of a set of vectors 
and a set of scalars that is closed under vector addition and scalar multiplication and that satisfies 
the usual rules of arithmetic. We will learn some of the vocabulary and phrases of linear algebra, 
such as linear independence, span, basis and dimension. We will learn about the four fundamental 
subspaces of a matrix, the Gram-Schmidt process, orthogonal projection, and the matrix formulation 
of the least-squares problem of drawing a straight line to fit noisy data. 
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Lecture 16 


Vector spaces 


View this lecture on YouTube 


A vector space consists of a set of vectors and a set of scalars. Although vectors can be quite gen- 
eral, for the purpose of this course we will only consider vectors that are real column matrices, and 
scalars that are real numbers. 

For the set of vectors and scalars to form a vector space, the set of vectors must be closed under 
vector addition and scalar multiplication. That is, when you multiply any two vectors in the set by 
real numbers and add them, the resulting vector must still be in the set. 

As an example, consider the set of vectors consisting of all three-by-one matrices, and let u and 
v be two of these vectors. Let w = au + bv be the sum of these two vectors multiplied by the real 
numbers a and Bb. If w is still a three-by-one matrix, then this set of vectors is closed under scalar 
multiplication and vector addition, and is indeed a vector space. The proof is rather simple. If we let 


uy O71 
u=|u|, v=]|o], 
U3 03 
then 
au, + bv, 
w =au+ bv = | aun + bv 
au3 + bv3 


is evidently a three-by-one matrix. This vector space is called IR°. 

Our main interest in vector spaces is to determine the vector spaces associated with matrices. There 
are four fundamental vector spaces of an m-by-n matrix A. They are called the null space, the column 
space, the row space, and the left null space. We will meet these vector spaces in later lectures. 
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58 LECTURE 16. VECTOR SPACES 


Problems for Lecture 16 


1. Explain why the zero vector must be a member of every vector space. 

2. Explain why the following sets of three-by-one matrices (with real number scalars) are vector spaces: 
(a) The set of three-by-one matrices with zero in the first row; 
(b) The set of three-by-one matrices with first row equal to the second row; 
(c) The set of three-by-one matrices with first row a constant multiple of the third row. 


Solutions to the Problems 


Lecture 17 


Linear independence 


View this lecture on YouTube 
The vectors {uz,uz,...,Uy} are linearly independent if for any scalars c1, C2, ..., Cn, the equation 


cyUy + C2U2 +++ + CyUy, = 0 


has only the solution cy = cp = --+ = Cy = 0. What this means is that one is unable to write any of 


the vectors uj,U2,...,Un as a linear combination of any of the other vectors. For instance, if there was 
a solution to the above equation with c, ¢ 0, then we could solve that equation for u; in terms of the 
other vectors with nonzero coefficients. 


As an example consider whether the following three three-by-one column vectors are linearly 


independent: 
1 0 2 
u=]0}], v=J1], w=]3 
0 0 0 


Indeed, they are not linearly independent, that is, they are linearly dependent, because w can be written 
in terms of u and v. In fact, w = 2u + 3v. 


Now consider the three three-by-one column vectors given by 


1 0 0 
u=/0/], v= ]1/l, w=1]0 
0 0 1 


These three vectors are linearly independent because you cannot write any one of these vectors as a 
linear combination of the other two. If we go back to our definition of linear independence, we can 


see that the equation 


a 0 
aut+bv+cw= 1b] =10 
fa 0 


has as its only solution a = b = c = 0. 

For simple examples, visual inspection can often decide if a set of vectors are linearly independent. 
For a more algorithmic procedure, place the vectors as the rows of a matrix and compute the reduced 
row echelon form. If the last row becomes all zeros, then the vectors are linearly dependent, and if not 
all zeros, then they are linearly independent. 
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60 LECTURE 17. LINEAR INDEPENDENCE 


Problems for Lecture 17 


1. Which of the following sets of vectors are linearly independent? 


1 
(a) 1],/0],]}1 


0 1 
-1 dl i 
(b) Lial—the) 1 
1 1) \-1 
1 
()q J 1],] oO], 


Solutions to the Problems 


Lecture 18 


Span, basis and dimension 


View this lecture on YouTube 


Given a set of vectors, one can generate a vector space by forming all linear combinations of that 
set of vectors. The span of the set of vectors {v1,V2,...,Vn} is the vector space consisting of all linear 
combinations of v1,V2,...,Vn. We say that a set of vectors spans a vector space. 

For example, the set of vectors given by 


1 0 2 
O],)/1],]3 
0 0 0 


spans the vector space of all three-by-one matrices with zero in the third row. This vector space is a 
vector subspace of all three-by-one matrices. 

One doesn’t need all three of these vectors to span this vector subspace because any one of these 
vectors is linearly dependent on the other two. The smallest set of vectors needed to span a vector 
space forms a basis for that vector space. Here, given the set of vectors above, we can construct a basis 
for the vector subspace of all three-by-one matrices with zero in the third row by simply choosing two 
out of three vectors from the above spanning set. Three possible basis vectors are given by 


1 1 2 2 
i 1 y J 3 J 1 . 3 
0 0 0 0 


Although all three combinations form a basis for the vector subspace, the first combination is usually 
preferred because this is an orthonormal basis. The vectors in this basis are mutually orthogonal and 
of unit norm. 

The number of vectors in a basis gives the dimension of the vector space. Here, the dimension of 


the vector space of all three-by-one matrices with zero in the third row is two. 
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62 LECTURE 18. SPAN, BASIS AND DIMENSION 


Problems for Lecture 18 


1. Find an orthonormal basis for the vector space of all three-by-one matrices with first row equal to 
second row. What is the dimension of this vector space? 


Solutions to the Problems 


Practice quiz: Vector space definitions 


1. Which set of three-by-one matrices (with real number scalars) is not a vector space? 
a) The set of three-by-one matrices with zero in the second row. 
b) The set of three-by-one matrices with the sum of all rows equal to one. 
c) The set of three-by-one matrices with the first row equal to the third row. 
d) The set of three-by-one matrices with the first row equal to the sum of the second and third rows. 


2. Which one of the following sets of vectors is linearly independent? 


1 0 1 
a) O},};1],]-1 
0 0 0 
2 1 4 
b) 1],]-1], 6 
1 2 —2 
1 0 1 
c) 0}, 1],]-1 
—1 —1 0 
3 3 2 
d) 2),{)1],])1 
1 2 0 
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64 LECTURE 18. SPAN, BASIS AND DIMENSION 


3. Which one of the following is an orthonormal basis for the vector space of all three-by-one matrices 
with the sum of all rows equal to zero? 


ee a en 
AA Sel HKL i], 
v2\ og} v2 
ee ee 
WV) oP ale 

v2\ 5} ve\_, 
te beeh el 4 
C4 —e2t—-1L ly , 

2: 2. 2. 
v2\ gf v2\_,} v2\_, 
Pee ee eee ee 
v6\ 5} v6\_ j]' vel 


Solutions to the Practice quiz 


Lecture 19 


Gram-Schmidt process 


View this lecture on YouTube 


Given any basis for a vector space, we can use an algorithm called the Gram-Schmidt process to 
construct an orthonormal basis for that space. Let the vectors v1,V2,...,Vn be a basis for some n- 
dimensional vector space. We will assume here that these vectors are column matrices, but this process 
also applies more generally. 

We will construct an orthogonal basis uy, u2,...,U,, and then normalize each vector to obtain an 


orthonormal basis. First, define uj = v;. To find the next orthogonal basis vector, define 


Observe that uz is equal to vz minus the component of v2 that is parallel to u;. By multiplying both 
sides of this equation with Mt, it is easy to see that uluy = 0 so that these two vectors are orthogonal. 


The next orthogonal vector in the new basis can be found from 


(ulvs uy (ud v3) Up 


T TL 
uy uj, Uy ug 


u3 = V3 


Here, ug is equal to v3 minus the components of v3 that are parallel to u; and uz. We can continue in 
this fashion to construct n orthogonal basis vectors. These vectors can then be normalized via 


x uy 


= 1. 
(upu,)!/? 


Since ux, is a linear combination of v1,V2,...,V,, the vector subspace spanned by the first k basis 


vectors of the original vector space is the same as the subspace spanned by the first k orthonormal 


vectors generated through the Gram-Schmidt process. We can write this result as 


span{u;,uz,...,u,} = span{vy1,V2,..., VK}. 
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66 LECTURE 19. GRAM-SCHMIDT PROCESS 


Problems for Lecture 19 


1. Suppose the four basis vectors {v1,V2,V3, v4} are given, and one performs the Gram-Schmidt pro- 
cess on these vectors in order. Write down the equation to find the fourth orthogonal vector uy. Do 
not normalize. 


Solutions to the Problems 


Lecture 20 


Gram-Schmidt process example 


View this lecture on YouTube 


As an example of the Gram-Schmidt process, consider a subspace of three-by-one column matrices 


with the basis 


1 0 
{v1,v2} = 1], 1|>, 
1 1 


and construct an orthonormal basis for this subspace. Let uy = v1. Then uz is found from 


(ufve)uy 
ug = V2 — a 
uy Uy 
1 —2 
=) = é 1] = : 1 
> 3 ~ 3 
1 1 1 
Normalizing the two vectors, we obtain the orthonormal basis 
if) i ie 
1,02} = « —]1], —; 1 
{ 1 2} JS ' 4/6 


Notice that the initial two vectors v1 and v2 span the vector subspace of three-by-one column matri- 
ces for which the second and third rows are equal. Clearly, the orthonormal basis vectors constructed 


from the Gram-Schmidt process span the same subspace. 
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68 LECTURE 20. GRAM-SCHMIDT PROCESS EXAMPLE 


Problems for Lecture 20 


1. Consider the vector subspace of three-by-one column vectors with the third row equal to the nega- 
tive of the second row, and with the following given basis: 


0 1 
W= 1], 1 
—1 —1 


Use the Gram-Schmidt process to construct an orthonormal basis for this subspace. 


2. Consider a subspace of all four-by-one column vectors with the following basis: 


II 
See Se eS 
Se oS 
PrP CO oO 


Use the Gram-Schmidt process to construct an orthonormal basis for this subspace. 


Solutions to the Problems 


Practice quiz: Gram-Schmidt process 


T T T 
: (ujv4)uy  (U5sVv4)U2 (U3 V4) UZ 
1. In the fourth step of the Gram-Schmidt process, the vector ug = v4 i T a - 
u, uy Uy U2 U3 U3 


is always perpendicular to 
a) vy 
b) vo 
c) V3 


d) v4 


N 


1 1 
The Gram-Schmidt process applied to {v1,v2} = (') , ( ) results in 


oes 1 fi 1 1 
vas -f 5) 4( 3) 


b) {ti, 2} <a 
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70 LECTURE 20. GRAM-SCHMIDT PROCESS EXAMPLE 


0 
3. The Gram-Schmidt process applied to {v1,v2} = 1], 1 results in 
—1 —1 
) {t,t} = 4 :).4 
a) {0,02} = < — 
1, U2 73 
—1 
pee : 
1,2} = ¢ — — 
1, U2 V3 : We - 


1 : 1 ’ 
c) {ti1, 1} = V3 Va —1 


d) {t,t%)}={—~] 1 
1 


Solutions to the Practice quiz 


Lecture 21 


Null space 


View this lecture on YouTube 
The null space of a matrix A, which we denote as Null(A), is the vector space spanned by all column 


vectors x that satisfy the matrix equation 
Ax = 0. 


Clearly, if x and y are in the null space of A, then so is ax + by so that the null space is closed under 
vector addition and scalar multiplication. If the matrix A is m-by-n, then Null(A) is a vector subspace 
of all n-by-one column matrices. If A is a square invertible matrix, then Null(A) consists of just the 


zero vector. 


To find a basis for the null space of a noninvertible matrix, we bring A to reduced row echelon 


form. We demonstrate by example. Consider the three-by-five matrix given by 


—3 6 —1 1 -—7 t =2 2 3 -1 1 -2 2 3. -1 
1-2. 2 3 =1)-=] 33 6 —-1 1 -7)}—>1]0 0 5 10 —10] — 
2 —4 5 8 —4 2 —4 5 8 —4 0 0 1 2 —2 

Ll. =2 2 3. -1 1 -2 0 -1 3 

0 0 1 2 —-2})]—7 10 0 1 2 2 

0 0 5 10 —10 0 0 0 0 0 


We call the variables associated with the pivot columns, x; and x3, basic variables, and the variables 
associated with the non-pivot columns, x2, x4 and x5, free variables. Writing the basic variables on the 


left-hand side of the Ax = 0 equations, we have from the first and second rows 


xy = 2x9 + X4 — 3X5, 


x3 —2x4 + 2X5. 
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72 LECTURE 21. NULL SPACE 


Eliminating x; and x3, we can write the general solution for vectors in Null(A) as 


2x2 + X4 — 3x5 2 1 —3 
x2 1 0 0 

—2x4 + 2x5 =x2!/O0] +x.) —-2] +%5 2), 
X4 0 1 0 
x5 0 0 1 


where the free variables x2, x4, and x5 can take any values. By writing the null space in this form, a 


basis for Null(A) is made evident, and is given by 


2 1 —3 
1 0 0 
O|, 2], 2 
0 1 0 
0 0 1 


The null space of A is seen to be a three-dimensional subspace of all five-by-one column matrices. In 


general, the dimension of Null(A) is equal to the number of non-pivot columns of rref(A). 


Problems for Lecture 21 


1. Determine a basis for the null space of 


Solutions to the Problems 
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LECTURE 21. NULL SPACE 


Lecture 22 


Application of the null space 


View this lecture on YouTube 


An under-determined system of linear equations Ax = b with more unknowns than equations may 
not have a unique solution. If u is the general form of a vector in the null space of A, and v is any 
vector that satisfies Av = b, then x = u+v satisfies Ax = A(u+v) = Au+ Av = 0+b = b. The 
general solution of Ax = b can therefore be written as the sum of a general vector in Null(A) and a 
particular vector that satisfies the under-determined system. 

As an example, suppose we want to find the general solution to the linear system of two equations 
and three unknowns given by 


2x41 + 2x9 + x3 = 0, 


2x41 — 2x2 — x3 = 1, 


x1 
2 2 1 0 
x2 = : 
2 —2 -1 1 
x3 


We first bring the augmented matrix to reduced row echelon form: 


2 2 10) ,f1 0 0 1/4 
2 22 1 0 1 1/2 —-1/4) 


which in matrix form is given by 


The null space satisfying Au = 0 is determined from wu, = 0 and uy = —u3/2, and we can write 
0 
Null(A) = span ¢ | —1 
2 


A particular solution for the inhomogeneous system satisfying Av = b is found by solving v; = 1/4 
and v2 + v3/2 = —1/4. Here, we simply take the free variable v3 to be zero, and we find vj = 1/4 
and vz = —1/4. The general solution to the original underdetermined linear system is the sum of the 
null space and the particular solution and is given by 


x4 0 
—a 1 | 1 
x2 T 4 
x3 2 0 


76 LECTURE 22. APPLICATION OF THE NULL SPACE 


Problems for Lecture 22 


1. Find the general solution to the system of equations given by 


3x1 + 6x2 — x3 4x4 = —7, 
x1 2X2 2x3 3X4 = 1, 
2x1 — 4x2 + 5x3 + 8x4 = —A4. 


Solutions to the Problems 


Lecture 23 


Column space 


View this lecture on YouTube 


The column space of a matrix is the vector space spanned by the columns of the matrix. When a 


matrix is multiplied by a column vector, the resulting vector is in the column space of the matrix, as 


(* 3) (*)=(eri8) =:(*) +(8) 


In general, Ax is a linear combination of the columns of A. Given an m-by-n matrix A, what is the 


can be seen from 


dimension of the column space of A, and how do we find a basis? Note that since A has m rows, the 
column space of A is a subspace of all m-by-one column matrices. 


Fortunately, a basis for the column space of A can be found from rref(A). Consider the example 


i= © 21: 3 
A={ 1-2 2 3-1], wef(A)=[0 0 1 2 -2 
0 0 0 0 0 


The matrix equation Ax = 0 expresses the linear dependence of the columns of A, and row operations 
on A do not change the dependence relations. For example, the second column of A above is —2 times 
the first column, and after several row operations, the second column of rref(A) is still —2 times the 
first column. 

It should be self-evident that only the pivot columns of rref(A) are linearly independent, and the 
dimension of the column space of A is therefore equal to its number of pivot columns; here it is two. 


A basis for the column space is given by the first and third columns of A, (not rref(A)), and is 


—3 —1 
1], 
5 


Recall that the dimension of the null space is the number of non-pivot columns—equal to the 
number of free variables—so that the sum of the dimensions of the null space and the column space 
is equal to the total number of columns. A statement of this theorem is as follows. Let A be an m-by-n 


matrix. Then 


dim(Col(A)) + dim(Null(A)) = n. 
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Problems for Lecture 23 


1. Determine the dimension and find a basis for the column space of 
11140 
A=1/1 101 
10411 


Solutions to the Problems 


LECTURE 23. COLUMN SPACE 


Lecture 24 


Row space, left null space and rank 


View this lecture on YouTube 


In addition to the column space and the null space, a matrix A has two more vector spaces asso- 
ciated with it, namely the column space and null space of A!, which are called the row space and the 
left null space. 

If A is an m-by-n matrix, then the row space and the null space are subspaces of all n-by-one 
column matrices, and the column space and the left null space are subspaces of all m-by-one column 
matrices. 

The null space consists of all vectors x such that Ax = 0, that is, the null space is the set of all 
vectors that are orthogonal to the row space of A. We say that these two vector spaces are orthogonal. 

A basis for the row space of a matrix can be found from computing rref(A), and is found to be 
rows of rref(A) (written as column vectors) with pivot columns. The dimension of the row space of A 
is therefore equal to the number of pivot columns, while the dimension of the null space of A is equal 
to the number of nonpivot columns. The union of these two subspaces make up the vector space of all 
n-by-one matrices and we say that these subspaces are orthogonal complements of each other. 

Furthermore, the dimension of the column space of A is also equal to the number of pivot columns, 
so that the dimensions of the column space and the row space of a matrix are equal. We have 


dim(Col(A)) = dim(Row(A)). 


We call this dimension the rank of the matrix A. This is an amazing result since the column space and 
row space are subspaces of two different vector spaces. In general, we must have rank(A) < min(m,n). 
When the equality holds, we say that the matrix is of full rank. And when A is a square matrix and of 
full rank, then the dimension of the null space is zero and A is invertible. 
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Problems for Lecture 24 


1. Find a basis for the column space, row space, null space and left null space of the four-by-five 


matrix A, where 


1 -2 3 -1 -3 


Check to see that null space is the orthogonal complement of the row space, and the left null space is 
the orthogonal complement of the column space. Find rank(A). Is this matrix of full rank? 


Solutions to the Problems 


Practice quiz: Fundamental subspaces 


1201 
1. Which of the following sets of vectors form a basis for the null space of [ 4 1 ; 2 
3.611 


{G0 
ai) 
aE] 
aE) 
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2. The general solution to the system of equations given by 


X41 + 2x9 +X = 1, 


2x1 + 4%. +43 4%, =1, 


3x1 + 6%. +%34+%4 =1, 


is 


0 —2 
0 1 
aja + 
0 0 
1 0 
22 0 
1 
b) a + : 
0 0 
0 1 
0 0 
0 0 
c)a + 
0 —3 
1 2 
0 0 
d) a : + : 
3 0 
2 1 
120 1 
3. What is the rank of the matrix }2 4 1 11]? 
3611 
a) 1 
b) 2 
c) 3 
d) 4 


Solutions to the Practice quiz 


Lecture 25 


Orthogonal projections 


View this lecture on YouTube 


Suppose that V is the n-dimensional vector space of all n-by-one matrices and W is a p-dimensional 
subspace of V. Let {81,52,-..,Sp} be an orthonormal basis for W. Extending the basis for W, let 
{s1,82,.-. / Sp, ty, t,-- “pnp be an orthonormal basis for V. 

Any vector v in V can be expanded using the basis for V as 


V = Ay} + 4282 + +++ + ApSp 4 byt, + bot 4 bn—ptn—p, 
where the a’s and b’s are scalar coefficients. The orthogonal projection of v onto W is then defined as 
Vprojy — 4151 + 4989 +--°-+ ApSp, 


that is, the part of v that lies in W. 
If you only know the vector v and the orthonormal basis for W, then the orthogonal projection of 


v onto W can be computed from 
T T T 
Vprojy = (V81)81 + (V'S2)82 +--+ + (V'Sp)Sp, 
that is, aj = v's, a> = v's», ete. 
We can prove that the vector Vproj,, is the vector in W that is closest to v. Let w be any vector in W 
different than Vprojy and expand w in terms of the basis vectors for W: 
W = C18, + C282 + °° + CpSp- 
The distance between v and w is given by the norm ||v — w||, and we have 


IIv — wll? = (a1 — c1)? + (a2 — €2)? +++ + (ap — Cp)? FBT DR + + Bp 


= bp +b +--+: +0i_» = lv = Vprojy II, 


or ||V — Vprojy|| < ||v — w]|, a result that will be used later in the problem of least squares. 
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Problems for Lecture 25 


a 1 0 
1. Find the general orthogonal projection of v onto W, where v = | b | and W = span 1 
c 
1 
What are the projections when v = | 0 | and when v = 
0 


oO fF © 
~_ 


Solutions to the Problems 


Lecture 26 


The least-squares problem 


View this lecture on YouTube 


Suppose there is some experimental data that you want to fit by a straight line. This is called a 


linear regression problem and an illustrative example is shown below. 


x 
Linear regression 
In general, let the data consist of a set of n points given by (x1,41), (X2,Y2),---, (%n,/Yn). Here, we 
assume that the x values are exact, and the y values are noisy. We further assume that the best fit line 


to the data takes the form y = fo + 61x. Although we know that the line will not go through all of the 
data points, we can still write down the equations as if it does. We have 


yi = Bot Pix, Yo= Pot Pix2, ---, Yn = Bot Bixn- 


These equations constitute a system of n equations in the two unknowns fo and $;. The corresponding 


matrix equation is given by 


1 x4 YI 
1 Xx Bo\ _ | ¥2 
Dt (*") |: 
1 XxX Yn 


This is an overdetermined system of equations with no solution. The problem of least squares is to 
find the best solution. 

We can generalize this problem as follows. Suppose we are given a matrix equation, Ax = b, that 
has no solution because b is not in the column space of A. So instead we solve Ax = bproj 


Dprojcaia) 
solution for x. 


cotta)’ where 


is the projection of b onto the column space of A. The solution is then called the least-squares 
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Problems for Lecture 26 


1. Suppose we have data points given by (x;,y;) = (0,1), (1,3), (2,3), and (3,4). If the data is to be 
fit by the line y = Bo + 61x, write down the overdetermined matrix expression for the set of equations 
Yi = Bo + B1Xi- 


Solutions to the Problems 


Lecture 27 


Solution of the least-squares problem 


View this lecture on YouTube 


We want to find the least-squares solution to an overdetermined matrix equation Ax = b. We write 
b= Dprojcaay + (b — bproj Bate ), where Dprojcg, a) is the projection of b onto the column space of A. Since 
(b — Dproj as age is orthogonal to the column space of A, it is in the nullspace of ue Multiplication of 
the overdetermined matrix equation by AT then results in a solvable set of equations, called the normal 
equations for Ax = b, given by 

ATAx = Alb. 


A unique solution to this matrix equation exists when the columns of A are linearly independent. 
An interesting formula exists for the matrix which projects b onto the column space of A. Multi- 
plying the normal equations on the left by A(ATA)~!, we obtain 


Ax = A(ATA)“!A'b =b 


PYO}col(A)* 


Notice that the projection matrix P = A(ATA)~!AT satisfies P? = P, that is, two projections is the 
same as one. If A itself is a square invertible matrix, then P = I and b is already in the column space 
of A. 

As an example of the application of the normal equations, consider the toy least-squares problem of 
fitting a line through the three data points (1,1), (2,3) and (3,2). With the line given by y = Bo + Bix, 
the overdetermined system of equations is given by 


ti ; 1 
1 2] ()=[3 
1 3) 2 


The least-squares solution is determined by solving 


111\{1! 111\([1 
a) cA (y= 33) a 
1 3) 2 
3 6\(Bo\_ (6 
6 14) \Bi) \13)° 


We can using Gaussian elimination to determine Bg = 1 and 6; = 1/2, and the least-squares line is 


given by y = 1+ x/2. The graph of the data and the line is shown below. 
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3r e 4 

>2t a ae. 4 

1) ° 7 
1 2 3 


Solution of a toy least-squares problem. 
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Problems for Lecture 27 


1. Suppose we have data points given by (X,Y) = (0,1), (1,3), (2,3), and (3,4). By solving the 
normal equations, fit the data by the line y = Bo + fix. 


Solutions to the Problems 
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LECTURE 27. SOLUTION OF THE LEAST-SQUARES PROBLEM 


Practice quiz: Orthogonal projections 


0 0 —2 
1. Which vector is the orthogonal projection of v = q onto W = span | 1 : | : | 
1 -1 1 


2. Suppose we have data points given by (xn,Yn) = (1,1), (2,1), and (3,3). If the data is to be fit by 
the line y = Bo + B1x, which is the overdetermined equation for By and f1? 


(6-0 


0-0 
()@)-( 
(3)0)-() 
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3. Suppose we have data points given by (Xn,Y%n) = (1,1), (2,1), and (3,3). Which is the best fit line 
to the data? 


Ngan 
3 

b y=-3 tx 

y=1t yx 

a) y= 1-52 


Solutions to the Practice quiz 


Week IV 


Eigenvalues and Eigenvectors 


95 


In this week’s lectures, we will learn about determinants and the eigenvalue problem. We will 
learn how to compute determinants using a Laplace expansion, the Leibniz formula, or by row or 
column elimination. We will formulate the eigenvalue problem and learn how to find the eigenvalues 
and eigenvectors of a matrix. We will learn how to diagonalize a matrix using its eigenvalues and 


eigenvectors, and how this leads to an easy calculation of a matrix raised to a power. 
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Lecture 28 


Two-by-two and three-by-three 


determinants 


View this lecture on YouTube 


We already showed that a two-by-two matrix A is invertible when its determinant is nonzero, where 
a b 
det A = = ad — be. 
cd 


If A is invertible, then the equation Ax = b has the unique solution x = A7!b. But if A is not invertible, 
then Ax = b may have no solution or an infinite number of solutions. When det A = 0, we say that 
the matrix A is singular. 

It is also straightforward to define the determinant for a three-by-three matrix. We consider the 
system of equations Ax = 0 and determine the condition for which x = 0 is the only solution. With 


abe xy 
de fi |x| =0, 
g hi x3 


one can do the messy algebra of elimination to solve for x1, x2, and x3. One finds that x1 = x2 = x3 = 0 
is the only solution when det A ¥ 0, where the definition, apart from a constant, is given by 


det A = aei + bfg + cdh — ceg — bdi — afh. 


An easy way to remember this result is to mentally draw the following picture: 


abc a b abc aé b 
Oy Pe ee 
e de |} _]|deoe de 
+ & 7 # # 
g h i gih g h i gih 


The matrix A is periodically extended two columns to the right, drawn explicitly here but usually only 
imagined. Then the six terms comprising the determinant are made evident, with the lines slanting 
down towards the right getting the plus signs and the lines slanting down towards the left getting the 
minus signs. Unfortunately, this mnemonic only works for three-by-three matrices. 
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98 LECTURE 28. TWO-BY-TWO AND THREE-BY-THREE DETERMINANTS 


Problems for Lecture 28 

1. Find the determinant of the three-by-three identity matrix. 

2. Show that the three-by-three determinant changes sign when the first two rows are interchanged. 
3. Let A and B be two-by-two matrices. Prove by direct computation that det AB = det A det B. 


Solutions to the Problems 


Lecture 29 


Laplace expansion 


View this lecture on YouTube 


There is a way to write the three-by-three determinant that generalizes. It is called a Laplace expansion 
(also called a cofactor expansion or expansion by minors). For the three-by-three determinant, we have 


a 
d aei + bfg +cdh — ceg — bdi—afh 
§ 


re SC 
Sh 
lI 


= a(ei— fh) — b(di — fg) +c(dh — eg), 


abe d d 

de jj=«l, {|-» bar HE 
i i 

gh i & & 


Evidently, the three-by-three determinant can be computed from lower-order two-by-two determi- 
nants, called minors. The rule here for a general n-by-n matrix is that one goes across the first row of 
the matrix, multiplying each element in the row by the determinant of the matrix obtained by crossing 
out that element’s row and column, and adding the results with alternating signs. 

In fact, this expansion in minors can be done across any row or down any column. When the minor 
is obtained by deleting the ith-row and j-th column, then the sign of the term is given by (—1)'47 . An 
easy way to remember the signs is to form a checkerboard pattern, exhibited here for the three-by-three 


and four-by-four matrices: 


“F P= 
a) = SE 
—_ + ~ , 
Sine ion 
= 
Example: Compute the determinant of 
1 Oo oO -1 
i 3 0 O 5 
2 2 4 -3 
1 0 5 0 


We first expand in minors down the second column. The only nonzero contribution comes from the 
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two in the third row, and we cross out the second column and third row (and multiply by a minus 
sign) to obtain a three-by-three determinant: 


1 0 oO -1 
1 0 -1 

3 0 0 5 
=-2/3 0 5 

a 2 —3 
1 5 0 

1 0 5 O 


We then again expand in minors down the second column. The only nonzero contribution comes 
from the five in the third row, and we cross out the second column and third row (and mutiply by a 
minus sign) to obtain a two-by-two determinant, which we then compute: 


1 0 1 1 1 
—2/3 O 5}|=10 = 80. 
3. 5 
1 5 0 


The trick here is to expand by minors across the row or column containing the most zeros. 


Problems for Lecture 29 


1. Compute the determinant of 


Solutions to the Problems 


YNoOon fF N 


NON Fe 
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Lecture 30 


Leibniz formula 


View this lecture on YouTube 


Another way to generalize the three-by-three determinant is called the Leibniz formula, or more de- 
scriptively, the big formula. The three-by-three determinant can be written as 


a b 
de f|=aei—afh+bfg—bdi+ cdh — ceg, 
gih 


where each term in the formula contains a single element from each row and from each column. For 
example, to obtain the third term bfg, b comes from the first row and second column, f comes from 
the second row and third column, and g comes from the third row and first column. As we can choose 
one of three elements from the first row, then one of two elements from the second row, and only 
one element from the third row, there are 3! = 6 terms in the formula, and the general n-by-n matrix 
without any zero entries will have n! terms. 

The sign of each term depends on whether the choice of columns as we go down the rows is an even 
or odd permutation of the columns ordered as {1,2,3,...,n}. An even permutation is when columns 
are interchanged an even number of times, and an odd permutation is when they are interchanged an 
odd number of times. Even permutations get a plus sign and odd permutations get a minus sign. 

For the determinant of the three-by-three matrix, the plus terms aei, bfg, and cdh correspond to 
the column orderings {1,2,3}, {2,3,1}, and {3,1,2}, which are even permutations of {1,2,3}, and 
the minus terms afh, bdi, and ceg correspond to the column orderings {1,3,2}, {2,1,3}, and {3,2,1}, 


which are odd permutations. 
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Problems for Lecture 30 


1. Using the Leibniz formula, compute the determinant of the following four-by-four matrix: 


ono NS FS 
= ©: 
“ OO. 


oo f 82 


Solutions to the Problems 


Lecture 31 


Properties of a determinant 


View this lecture on YouTube 


The determinant is a function that maps a square matrix to a scalar. It is uniquely defined by the 
following three properties: 

Property 1: The determinant of the identity matrix is one; 

Property 2: The determinant changes sign under row interchange; 

Property 3: The determinant is a linear function of the first row, holding all other rows fixed. 


Using two-by-two matrices, the first two properties are illustrated by 


1 0 LA and a b a c a 
01 c a b 
and the third property is illustrated by 
ka kb ab a+a’ b+bd' a b a’ oO! 
— and = . 
c h(Ud cfd Cc d cfd c h(Ud 


Both the Laplace expansion and Leibniz formula for the determinant can be proved from these three 
properties. Other useful properties of the determinant can also be proved: 


e The determinant is a linear function of any row, holding all other rows fixed; 

e If a matrix has two equal rows, then the determinant is zero; 

e If we add k times row-i to row-j, the determinant doesn’t change; 

e The determinant of a matrix with a row of zeros is zero; 

e A matrix with a zero determinant is not invertible; 

e The determinant of a diagonal matrix is the product of the diagonal elements; 

e The determinant of an upper or lower triangular matrix is the product of the diagonal elements; 
e The determinant of the product of two matrices is equal to the product of the determinants; 

e The determinant of the inverse matrix is equal to the reciprical of the determinant; 


e The determinant of the transpose of a matrix is equal to the determinant of the matrix. 


Notably, these properties imply that Gaussian elimination, done on rows or columns or both, can be 
used to simplify the computation of a determinant. Row interchanges and multiplication of a row by 
a constant change the determinant and must be treated correctly. 
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Problems for Lecture 31 


1. Using the defining properties of a determinant, prove that if a matrix has two equal rows, then the 


determinant is zero. 


2. Using the defining properties of a determinant, prove that the determinant is a linear function of 


any row, holding all other rows fixed. 


3. Using the results of the above problems, prove that if we add k times row-i to row-j, the determinant 


doesn’t change. 


4. Use Gaussian elimination to find the determinant of the following matrix: 


2 0 -1 
A=1]3 1 1 
0 -1 


Solutions to the Problems 


Practice quiz: Determinants 


-3 0-2 O O 
—2 —2 0 0 
1. The determinant of 0 O -2 O- Of} is equal to 
0-3 2 -3 
-3 3 3 O -2 
a) 48 
b) 42 
c) —42 
d) —48 
aeO 0 
b 
2. The determinant of fs : is equal to 
c Oh i 
d 0 0 j 


a) afhj + behj — cegj — degi 
b) afhj — behj + cegj — degi 
c) agij — beij +cefj—defh 
d) agij + beij — cefj —defh 
3. Assume A and B are invertible n-by-n matrices. Which of the following identities is false? 
a) detA~! =1/detA 
b) det AT = detA 
c) det(A +B) = detA + detB 
d) det (AB) = det AdetB 


Solutions to the Practice quiz 
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Lecture 32 


The eigenvalue problem 


View this lecture on YouTube 


Let A be a square matrix, x a column vector, and A a scalar. The eigenvalue problem for A solves 
Ax = Ax 


for eigenvalues A; with corresponding eigenvectors x;. Making use of the identity matrix I, the eigen- 
value problem can be rewritten as 
(A — AI)x =0, 


where the matrix (A — AI) is just the matrix A with A subtracted from its diagonal. For there to be 


nonzero eigenvectors, the matrix (A — AI) must be singular, that is, 
det (A — AI) = 0. 


This equation is called the characteristic equation of the matrix A. From the Leibniz formula, the char- 
acteristic equation of an n-by-n matrix is an n-th order polynomial equation in A. For each found Aj;, a 
corresponding eigenvector x; can be determined directly by solving (A — A;I)x = 0 for x. 

For illustration, we compute the eigenvalues of a general two-by-two matrix. We have 


a—A b 
d—2X 


0 = det (A— Al) = = (a—A)(d—A) —be = A? — (a+ d)A+4 (ad — be); 


and this characteristic equation can be rewritten as 
M2 -—TrAA+detA =0, 


where Tr A is the trace, or sum of the diagonal elements, of the matrix A. 

Since the characteristic equation of a two-by-two matrix is a quadratic equation, it can have either 
(i) two distinct real roots; (ii) two distinct complex conjugate roots; or (iii) one degenerate real root. 
More generally, eigenvalues can be real or complex, and an n-by-n matrix may have less than n distinct 


eigenvalues. 
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Problems for Lecture 32 


1. Using the formula for a three-by-three determinant, determine the characteristic equation for a 
general three-by-three matrix A. This equation should be written as a cubic equation in A. 


Solutions to the Problems 


Lecture 33 


Finding eigenvalues and eigenvectors 


(1) 


View this lecture on YouTube 


We compute here the two real eigenvalues and eigenvectors of a two-by-two matrix. 
0 1 


Example: Find the eigenvalues and eigenvectors of A = (' al 


The characteristic equation of A is given by 
\*-1=0, 
with solutions A; = 1 and Az = —1. The first eigenvector is found by solving (A — A,I)x = 0, or 
—1 1 xy _0. 
1 -1 x2 


The equation from the second row is just a constant multiple of the equation from the first row and 
this will always be the case for two-by-two matrices. From the first row, say, we find x2 = x1. The 


second eigenvector is found by solving (A — AgI)x = 0, or 


so that x2 = —x,. The eigenvalues and eigenvectors are therefore given by 


Ay, =1 = ; Ay =-1 = ; 
=1x = 5 = , xo= : 
1 1 1 2 2 1 


The eigenvectors can be multiplied by an arbitrary nonzero constant. Notice that A; + Az = TrA and 
that AyA2 = det A, and analogous relations are true for any n-by-n matrix. In particular, comparing 


the sum over all the eigenvalues and the matrix trace provides a simple algebra check. 
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112 LECTURE 33. FINDING EIGENVALUES AND EIGENVECTORS (1) 
Problems for Lecture 33 


2-7 
1. Find the eigenvalues and eigenvectors of (; i F 


2. Find the eigenvalues and eigenvectors of 


> 

ll 
OrrRN 
PNR 
NFO 


Solutions to the Problems 


Lecture 34 


Finding eigenvalues and eigenvectors 
(2) 
View this lecture on YouTube 


We compute some more eigenvalues and eigenvectors. 


1 
Example: Find the eigenvalues and eigenvectors of B = ft ) : 


The characteristic equation of B is given by 
A? =0, 


so that there is a degenerate eigenvalue of zero. The eigenvector associated with the zero eigenvalue 
is found from Bx = 0 and has zero second component. This matrix therefore has only one eigenvalue 


A=0, «= (3). 
0 


-1 
Example: Find the eigenvalues of C = : ) 


and eigenvector, given by 


0 
The characteristic equation of C is given by 


4+1=0, 


which has the imaginary solutions 1 = +i. Matrices with complex eigenvalues play an important role 


in the theory of linear differential equations. 
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114 LECTURE 34. FINDING EIGENVALUES AND EIGENVECTORS (2) 
Problems for Lecture 34 


1 1 
1. Find the eigenvalues of ( ; i) 


Solutions to the Problems 


Practice quiz: The eigenvalue problem 


1 -1 
1. Which of the following are the eigenvalues of ( ; ) 


2. 
3.4/3 
a) ae a 
2 2 
3 5 
je 
2 2 
1 V3 
c) a ere 
2: 2. 
1 5 
free 
2: 2 
F ; ; 3-1 
2. Which of the following are the eigenvalues of ( @. 
a) 1+3i 
b) 14+V3 
c) 3V3 +1 
d) 341 
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a) 10 
1 
1 
b) | /2 
1 
0 
c) } 1 
V2 
d) 1 
J2 


Solutions to the Practice quiz 


Lecture 35 


Matrix diagonalization 


View this lecture on YouTube 


For concreteness, consider a two-by-two matrix A with eigenvalues and eigenvectors given by 


X11 X12 
Ay X= ; Ax X= : 
X21 X22 


And consider the matrix product and factorization given by 


A{* *12) _ Aixi1  A2x12 _(%1 2) (Ar 0 
X21 X22 AyXxX21 A2x22 Xo X22 O Ap 


Generalizing, we define S to be the matrix whose columns are the eigenvectors of A, and A to be 


the diagonal matrix with eigenvalues down the diagonal. Then for any n-by-n matrix with n linearly 


independent eigenvectors, we have 
AS =SA, 


where S is an invertible matrix. Multiplying both sides on the right or the left by S—!, we derive the 


relations 
A=SAS"! or A=S71AS. 


To remember the order of the S and S~! matrices in these formulas, just remember that A should be 


multiplied on the right by the eigenvectors placed in the columns of S. 
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Problems for Lecture 35 


1. Prove that two eigenvectors corresponding to distinct eigenvalues are linearly independent. 


2. Prove that if the columns of an n-by-n matrix are linearly independent, then the matrix is invertible. 
(An n-by-n matrix whose columns are eigenvectors corresponding to distinct eigenvalues is therefore 
invertible.) 


Solutions to the Problems 


Lecture 36 


Matrix diagonalization example 


View this lecture on YouTube 


ba 
The eigenvalues of A are determined from 


b 
Example: Diagonalize the matrix A = (; ) 


a—A b 


det(A — AI) = i . 
a pan 


Sls ei 


Solving for A, the two eigenvalues are given by Aj = a+b and Az = a—b. The corresponding 
eigenvector for A; is found from (A — A,I)x; = 0, or 


—b b X41 = 0 . 
b SB) Vea VOR” 
and the corresponding eigenvector for Az is found from (A — A2I)x2 = 0, or 
b b x42 _ 0 
b b}) \xxn}  \o}° 
Solving for the eigenvectors and normalizing them, the eigenvalues and eigenvectors are given by 


1 1 
Ay =a+b, a= J5(t): Az =a-—b, = F5( 7 


The matrix S of eigenvectors can be seen to be orthogonal so that S-1 = ST. We then have 


= 5 (i 4 and S-!=s'=sS; 
1 -1 


and the diagonalization result is given by 


fo ee Gay) 
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120 LECTURE 36. MATRIX DIAGONALIZATION EXAMPLE 


Problems for Lecture 36 


1. Diagonalize the matrix A = 


OFrRN 
BNR 
NFO 


Solutions to the Problems 


Lecture 37 


Powers of a matrix 


View this lecture on YouTube 


Diagonalizing a matrix facilitates finding powers of that matrix. Suppose that A is diagonalizable, 


and consider 
A? = (SAS~')(SAS~!) = SA2S“1, 


where in the two-by-two example, A? is simply 


A, O\ fA. 0O\_ fA? oO 
0 As} \O Ag ao way 
In general, A? has the eigenvalues raised to the power of p down the diagonal, and 


AP =SAPS"!. 
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Problems for Lecture 37 


1. From calculus, the exponential function is sometimes defined from the power series 


If A is diagonalizable, show that 


where 
e1 0 0 
0 e& 0 
c= . 
0 0 en 


Solutions to the Problems 


Lecture 38 


Powers of a matrix example 


View this lecture on YouTube 


n 


a b 
Example: Determine a general formula for (; , where n is a positive integer. 
a 


We have previously determined that the matrix can be written as 


Pete lew ae, 


Raising the matrix to the nth power, we obtain 


a b\’ 1/1 1\ ((a+by" 0 1 
Be. 20 al 0 (@—by"} \1 1 


And multiplying the matrices, we obtain 


(; ) | ts +b)"+(a—b)" (a+b)"—(a—b) 
(a+b)"—(a—b)" (a+b)"+(a—b)" 
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124 LECTURE 38. POWERS OF A MATRIX EXAMPLE 


Problems for Lecture 38 


n 
1 -1 
1. Determine ( ‘i , , where n is a positive integer. 


Solutions to the Problems 


Practice quiz: Matrix diagonalization 


1. Let A; and Az be distinct eigenvalues of a two-by-two matrix A. Which of the following cannot be 


the associated eigenvectors? 


(0 
rae(o-() 
ro(u-() 
ra-(Je( 


* 


a) 


ila 
(4) 
01 
(04) 
10 
o( 4) 
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126 LECTURE 38. POWERS OF A MATRIX EXAMPLE 


3. Which matrix is equal to e!, where I is the two-by-two identity matrix? 


ott) 


Solutions to the Practice quiz 


Appendix 
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Appendix A 


Problem and practice quiz solutions 


Solutions to the Problems for Lecture 1 


oOo rF Oo 


or CO oC 
eee 
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130 APPENDIX A. PROBLEM AND PRACTICE QUIZ SOLUTIONS 


Solutions to the Problems for Lecture 2 


am ’ , 3C—E: notdefined, AC: not defined, 


CD = 11 10 | CB= 8 —10 —3 
10 11 10 —-8 0 


23 4 2, 26 2. 
3. AD=]2 6 12}, DA=|]3 6 9 
2 9 16 4 12 16 


n P Pon P 
4. [A(BC)] jj = Del BC =) Vagbucy = De de ainbiaey = )C[AB] 1c; = [(AB)C)];. 


k=11=1 1=1 
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Solutions to the Problems for Lecture 3 


1. In general, matrix multiplication can be written as 


—1 2 
Let A = z . Notice that the second column of A is —2 times the first column. Then AB will 


be zero if the first row of B is 2 times the second row. For example, we have 
—1 2: 2 4\ /0 0 
A 28) VI, Bp kOe 0g 


2. Let A be an m-by-p diagonal matrix, B a p-by-n diagonal matrix, and let C = AB. The ij element of 
C is given by 


Pr 
Cij = x Aix Dyj- 
k=] 


Since A is a diagonal matrix, the only nonzero term in the sum is k = 7 and we have cj; = ajjbj;. And 


since B is a diagonal matrix, the only nonzero elements of C are the diagonal elements cjj = ajibji. 


3. Let A and B be n-by-n upper triangular matrices, and let C = AB. The ij element of C is given by 
n 
Cij = x Aix dg; 
k=1 


Since A and B are upper triangular, we have a; = 0 when k <i and by; = 0 when k > j. Excluding the 


zero terms from the summation, we have 


j 
te=) tbe 
ki 


which is equal to zero when i > j proving that C is upper triangular. Furthermore, cjj = ajjbjj. 


132 APPENDIX A. PROBLEM AND PRACTICE QUIZ SOLUTIONS 


Solutions to the Practice quiz: Matrix definitions 


. . = 
1. d. With aij =i-—j,we have a4, = 422 = 0, aq2 = —1, and az, = 1. Therefore A = é ) 7 


3. b. For upper triangular matrices A and B, aj, = 0 when k <i and bk; = 0 when k > j. 
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Solutions to the Problems for Lecture 4 


1. Let A be an m-by-p matrix, B a p-by-n matrix, and C = AB an m-by-n matrix. We have 
T E P T av 
Ci = Chi = s Aik Dk = ss big Ay- 
k=1 k=1 
With C! = (AB)!, we have proved that (AB)? = BTAT. 


2. The square matrix A + A! is symmetric, and the square matrix A — A! is skew symmetric. Using 


these two matrices, we can write 


A=5(AtAT)4+5(A At). 


3. Let A be a m-by-n matrix. Then using (AB)? = BTAT and (A™)! = A, we have 


(ATA)? = ATA. 
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Solutions to the Problems for Lecture 5 


b2 + c? 


APPENDIX A. PROBLEM AND PRACTICE QUIZ SOLUTIONS 


d 

a 
1. ad + be+cf 
i 


2. Let A be an m-by-n matrix. Then 


nom 


ad + be +cf 


n m nN 
TATA) = VATA) = LD aia = VY ai 
j=l ma 


jHli=l1 


which is the sum of the squares of all the elements of A. 


i=1j=1 
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Solutions to the Problems for Lecture 6 
4 -1 
5 6 5 -6 6 4 1 3-4 
1. = and —— . 
Ge) =(E SaaS) -a(3 


2. From the definition of an inverse, 
(AB)~1(AB) =1. 


Multiply on the right by B~!, and then by A7|, to obtain 


(AB) }=B TAT. 


3. We assume that A is invertible so that 
AA!=I1 and ATA=L 


Taking the transpose of both sides of these two equations, using both I’ = I and (AB)’ = B'AT, we 


obtain 
AA aA“ AST SI and (A “Ay aA A al SL 


Therefore, since we have found that 
(A)TAT=I and ATAT)TH=L 


we conclude that A’ is invertible and that (AT)~! = (A7!)?. 


4. Let A be an invertible matrix, and suppose B and C are its inverse. To prove that B = C, we write 


B = BI = B(AC) = (BA)C =C. 


136 APPENDIX A. PROBLEM AND PRACTICE QUIZ SOLUTIONS 
Solutions to the Practice quiz: Transpose and inverses 


1. d. (ABC)? = ((AB)C)? = CT(AB)? = CTBTAT. 


2. c. A symmetric matrix C satisfies C! =C. We can test all four matrices. 
(A+ AT)JT=AT+A=A+AT; 


(AAT)? = AAT; 

(A—AT)T=AT-A=—(A-—AY}); 

(ATA)? = ATA. 

Only the third matrix is not symmetric. It is a skew-symmetric matrix, where cl = —-c. 


3. a. Exchange the diagonal elements, negate the off-diagonal elements, and divide by the determinant. 


-1 
We have nie at Se : 
1 2 2\-1 2 


Solutions to the Problems for Lecture 7 


1. Let Q; and Q» be orthogonal matrices. Then 


(Q1Q2)7' = Q5'Q71 = QFQT = (QiQ)*. 


2. Since II = I, we have I~! = I. And since I? = I, we have I~! = I! and Tis an orthogonal matrix. 
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138 APPENDIX A. PROBLEM AND PRACTICE QUIZ SOLUTIONS 


Solutions to the Problems for Lecture 8 


a Ri-) = ( cos 6 -) = Re)“, 


—sin@ cosé 


2. The z-coordinate stays fixed, and the vector rotates an angle @ in the x-y plane. Therefore, 


cos@ —sind 0 
R, = | sin@ cos 6 0 
0 0 1 
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Solutions to the Problems for Lecture 9 


1; 
10 0 0 0 01 0 
Pi3=]0 1 Of], Pix2=]0 0 1], Pop=]1 0], 
001 0 1 0 0 0 
0 0 0 0 0 
P31 = | 0 1}, Ps32=]1 O}, P317=]0 1 0 
1 00 0 1 0 0 0 
2: 


a4 -1 4 -1 
Pio3 = Piz3,  Py39 = Paz2, Pay3 = Pais, P32, = Pasar, 
Py =P. Pp; =P. 
931 — * 3i2r 312 — * 231: 


The matrices that are their own inverses correspond to either no permutation or a single permutation 
of rows (or columns), e.g., {1,3,2}, which permutes row (column) two and three. The matrices that are 
not their own inverses correspond to two permutations, e.g., {2,3,1}, which permutes row (column) 


one and two, and then two and three. For example, commuting rows by left multiplication, we have 
Po31 = Py32Po13, 
so that the inverse matrix is given by 
Post = Pog Pign = PoiaPiso.- 


Because matrices in general do not commute, Pra # Po31. Note also that the permutation matrices 
are orthogonal, so that the inverse matrices are equal to the transpose matrices. Therefore, only the 


symmetric permutation matrices can be their own inverses. 


140 APPENDIX A. PROBLEM AND PRACTICE QUIZ SOLUTIONS 
Solutions to the Practice quiz: Orthogonal matrices 


1. d. An orthogonal matrix has orthonormal rows and columns. The rows and columns of the matrix 


1 -1 
( ) are not orthonormal and therefore this matrix is not an orthogonal matrix. 


2. a. The rotation matrix representing a counterclockwise rotation around the x-axis in the y-z plane 
can be obtained from the rotation matrix representing a counterclockwise rotation around the z-axis in 


the x-y plane by shifting the elements to the right one column and down one row, assuming a periodic 


1 0 0 
extension of the matrix. The result is | 0 cos@ —sin@ 
0 sin@é cos 0 


1 0 0 001 
3. b. Interchange the rows of the identity matrix: |}0 1 0/711 0 O 
001 01 0 


Solutions to the Problems for Lecture 10 


1. 


(a) Row reduction of the augmented matrix proceeds as follows: 


3. -7 -—2 —-7 3. -7 -2 —-7 
—3 5 1 5 > 10 2 1 2 
6 —4 0 2 0 10 4 16 


Solution by back substitution is given by 


x3 = —6, 
1 
= _—— =? =4, 
x2 5 (3 ) 


1 
x= 3 (7%2 + 2x3 — 7) = 3. 


The solution is therefore 


x4 3 
x2 = 
x3 —6 


1 —2 3 1 1 -2 3 1 
—1 3-1 -1]—-1]0 1 2 Oj - 
2 —5 o 1 0 -1 -1 -1 


Solution by back substitution is given by 


x%3= —1, 
x2= —2x3 = 2; 


xy = 2x9 -—3x3+1=8. 


The solution is therefore 


142 APPENDIX A. PROBLEM AND PRACTICE QUIZ SOLUTIONS 
Solutions to the Problems for Lecture 11 


1. 


(a) Row reduction proceeds as follows: 


3 -7 -2 —-7 3-7 -2 —-7 3 0 3/2 0 
A=]|-3 5 il 5 >10 2 1 2. > 10 2 1 2 > 
6 —4 0 2 0 10 4 16 0 0 -1 6 
3 0 0 9 1 0 0 3 
0 -2 0 -8]}—->1]0 dl 
0 0 -1 6 0 1 -6 
Here, columns one, two, and three are pivot columns. 
(b) Row reduction proceeds as follows: 
12 1 1 2 1 1 2 1 12 0 
A=/]2 4 1]}/—>]0 0 -1]—->]0 0O 1]/—-]0 0 1 
3 6 2 00 -1 0 0 -1 0 0 0 


Here, columns one and three are pivot columns. 


Solutions to the Problems for Lecture 12 


1; 
3 -7 —2 10 0 3-7 -2 100 
—3 5 1031 0;]-> 0 —2 -!1 110]-> 
6 —4 0001 0 10 4 —-2 0 1 
3 0 3/2 -5/2 —7/2 0 3 0 3/2 -—5/2 -—7/2 0 
0 —2 -!1 1 1 0j;/—-]0 -—2 -1 1 1 0 
0 0 -1 3 5 1 0 0 -1 3 5 1 
1 0 0 2/3 4/3 1/2 
010 41 2 1/2 
001 -3 -5 -!1 

Therefore, 


a a a 2/3 4/3 1/2 
= 6. 4 = 1 2 1/2 


144 APPENDIX A. PROBLEM AND PRACTICE QUIZ SOLUTIONS 


Solutions to the Practice quiz: Gaussian elimination 


1 -2 1 0 1 -2 1 0 1 -2 1 0 1 -2 1 0 
lia. | 2 1 -3 5| > ]0 5 —5 5] ]0 1 -1 1}/— 10 1 -1 1 
4 —-7 1, =2 0 1 -3 -2 0 1 =3 = 2 0 0 -—2 -3 


2. c. A matrix in reduced row echelon form has all its pivots equal to one, and all the entries 
above and below the pivots eliminated. The only matrix that is not in reduced row echelon form is 
101 0 
0 1 0 OJ. The pivot in the third row, third column has a one above it in the first row, third 


0011 
column. 


3. d. There are many ways to do this computation by hand, and here is one way: 


3 -7 -—2 1 0 0 3-7 —2 1 0 0 3 -7 -—2 1 0 0 

3 =#=5 101 0;—741]0 -2 -1 1 10);74;];0 -2 -1 11 0/7 

6-4 0001 0 10 4 —2 0 1 0 0 -1 3 5 1 

3 -7 0 -—5 —10 —2 3 7 0 -5 10 —2 3.00 2 4 3/2 

0 -—2 -1 1 1 0 >10 -2 0 -2 4 il >1/0 1 0 1 2 1/2) > 
0 0 f) -3 =)» ssl 0 01 -3 -5 -!I1 001 -3 -5 -1 

1 0 0 2/3 4/3 1/2 3° 7 = 2 - 2/3 4/3 1/2 

0 1 0 1 2 1/2] . Therefore, [ 5 1 = 1 2 1/2 

001 -3 -5 -!1 6 —-4 0 -3 -5 -1 


Solutions to the Problems for Lecture 13 


oo lO le 


Nu OF © 


or oO CO 


pep oOo Oo O&O 
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146 APPENDIX A. PROBLEM AND PRACTICE QUIZ SOLUTIONS 


Solutions to the Problems for Lecture 14 


1; 
3-7 -2 3-7 —2 10 0 3-7 -—2 
—3 5 1] -> 0 —2 -1}/=/1 10 —3 5 1 
6 —4 0 6 —4 0 001 6 —4 0 
3-7 -2 3-7 —2 10 0 3-7 -—2 
0 2 1 >10 2 1] = 01 0 0 -—2 -1 
6 —4 0 0 10 4 —2 0 1 6 —4 0 
3-7 —2 3-7 -2 10 0 3-7 -2 
0 2 1 > 10 2 1/=]0 1 0 0 -—2 -1 
0 10 4 0 0 -i1 05 1 0 10 4 
Therefore, 
3-7 —2 1 0 0 3-7 —2 
—3 5 1} =]-1 if 0 0 -—2 -1 


Solutions to the Problems for Lecture 15 


1. We know 
3 -7 -2 1 0 0 3 -7 -—2 
A=|-3 5 1/=]-1 1 0 0 —2 -1] =LU 
6 —4 0 2 —5 1 0 0 —-1 


To solve LUx = b, we let y = Ux, solve Ly = b for y, and then solve Ux = y for x. 
(a) 


The equations Ly = b are given by 


y1 = —3, 
—yit+y2 =3, 
2y1 = 5y2 + ¥3 = 2, 


with solution y; = —3, y2 = 0, and y3 = 8. The equations Ux = y are given by 


3x1 — 7x9 — 2x3 = —3 
—2x2 — x3 =0 


7x3 = 8, 


with solution x3 = —8, x» = 4, and x; = 3. 


(b) 


The equations Ly = b are given by 


y=1, 
Vr y= =i 
2y1 — 5y2 +93 = 1, 


with solution y; = 1, y2 = 0, and y3 = —1. The equations Ux = y are given by 


3x1 — 7x2 — 2x3 =1 
—2x — x3 =0 


7x3 = -1L 


with solution x3 = 1, x2 = —1/2, and x, = —1/6. 


148 APPENDIX A. PROBLEM AND PRACTICE QUIZ SOLUTIONS 
Solutions to the Practice quiz: LU decomposition 


1. c. Start with the identity matrix. In the third row (changed row) and second column (the row which 


0 0 0 
; vais ae 1 0 0 
is multiplied by 2) , place a 2. The elementary matrix is ae a 
000 1 
2. b. 
3. -7 -2 3. -7 -2 1 0 
A=1|-3 5 1 >10 2 1] =M,A, whereM,;=]1 1 O]; 
6 -4 0 6 -4 0 00 1 
3. -7 —2 3-7 -2 1 0 0 
0 2 1 >10 2 1} =MoM,A, where My = 0 1 OF; 
6 -4 0 0 10 4 —2 0 1 
3-7 -2 3 .-7 -2 10 O 
0 2 i > 10 2 1] =M3MoM,A, where M3 = 01 0 
0 10 4 0 0 -1 0 5 1 
Therefore, 


N 
| 
ol 
_ 
S 
Oo 
| 
an 


3. b. To solve LUx = b, let y = Ux. Then solve Ly = b for y and Ux = y for x. The equations given by 


Ly = bare 
y=1, 
=H rye = -1, 
2y1 — 5y2 + y3 = 1. 
1 
Solution by forward substitution gives y = 0 |. The equations given by Ux = y are 
—1 
3x1 — 7x2 — 2x3 = 1, 
—2Xx9 — x= 0, 
—x%3>= —1. 
—1/6 
Solution by backward substitution gives x = | —1/2 


1 
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Solutions to the Problems for Lecture 16 


1. Let v be a vector in the vector space. Then both Ov and v + (—1)v must be vectors in the vector 


space and both of them are the zero vector. 


2. In all of the examples, the vector spaces are closed under scalar multiplication and vector addition. 


150 APPENDIX A. PROBLEM AND PRACTICE QUIZ SOLUTIONS 
Solutions to the Problems for Lecture 17 


1. 


(a) We place the vectors as the rows of a matrix and compute the reduced row echelon form: 


1 1 0 1 1 0 1 0 1 1 0 0 
1 01}]/—74 70 -1 1] 710 1 -1/—-]0 1 0 
011 0 1 1 0 0 2 0 0 1 


Because the reduced row echelon form is the identity matrix, the vectors are linearly indepen- 
dent. 


(b) We place the vectors as the rows of a matrix and compute the reduced row echelon form: 


=1 1 1 1 -1 -1 1 -1 -1 1 0 0 
1 -1 1] ]0 0 2} 10 1 0;—]0 1 0 
1 1 -1 0 2 0 0 0 1 0 0 1 


Because the reduced row echelon form is the identity matrix, the vectors are linearly indepen- 
dent. 


(c) By inspection, 


The vectors are therefore linearly dependent. 


Solutions to the Problems for Lecture 18 


1. One possible orthonormal basis is 


The dimension of this vector space is two. 
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152 APPENDIX A. PROBLEM AND PRACTICE QUIZ SOLUTIONS 
Solutions to the Practice quiz: Vector space definitions 


1. b. A vector space must be closed under vector addition and scalar multiplication. The set of three- 
by-one matrices with the sum of all rows equal to one is not a closed set. For example, k times a vector 


whose sum of all rows is equal to one results in a vector whose sum of all rows is equal to k. 


2. d. One can find the relations 


1 1 1 1 0 
O}—{1]=]-1], 8] -1]+3 6} =10] 1], O;-—]-1]= 1], 
0 0 2 —2 —1 0 —1 


so that these sets of three matrices are linearly dependent. The remaining set of vectors can be put in 
the rows of a matrix and the reduced row echelon form can be computed: 


3.92 1 3 2 1 3 0 38 1 0 0 
3 1 2}/71/0 -1 1 > 10 1 -1]/—>]0 1 0 
2 1 0 0 -1/3 —2/3 0 oO -!1 0 0 1 


Because the reduced row echelon form is the identity matrix, the vectors are linearly independent. 


3. b. Since a three-by-one matrix has three degrees of freedom, and the constraint that the sum 
of all rows equals zero eliminates one degree of freedom, the basis should consist of two vectors. 


1 

We can arbitrarily take the first unnormalized vector to be | —1]. The vector orthogonal to this 
0 

first vector with sum of all rows equal to zero is 1 |. Normalizing both of these vectors, we get 


—2 


Solutions to the Problems for Lecture 19 


1. 
(ulva)u; (usva)u2 (udv4)ug 
ug = V4 T T T < 
u, Uy Uy U2 U3 U3 
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154 APPENDIX A. PROBLEM AND PRACTICE QUIZ SOLUTIONS 


Solutions to the Problems for Lecture 20 


1. Define 


{vi,v2} = 1 y 1 


Let uy = vj. Then up is found from 


_ (ulvo)uy 
u2 = V2 — T 
u, uy 
1 
—1 —1 0 


Normalizing, we obtain the orthonormal basis 


1 
1 
{ti1,t2}={—=] 1],]0 
2 
W2 —1 0 
2. Define 
1 0 0 
1 1 0 
Vv ,V' 7V = y ¥ 
wea a ae ee 
1 1 1 
Let u, = vj. Then up is found from 
(uf v2)uy 
ug = V2 — a Sn 
Uy uj, 
0 1 —3 
fi 3)1}) 1 1 
tay} 4fay i] 
1 1 1 


and u3 is found from 


= (utv3)uy (ujv3)u2 
u3 = V3 T T 
u, Uy uy U2 


il >) 
me 
pe eS 
m 
me 
| 
N 


Normalizing the three vectors, we obtain the orthonormal basis 


1 3 
ware i)! 3 1 
uy,,U2,U = , aT aod 
{ti tip, ts} 4) 993.) <4 
1 
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156 APPENDIX A. PROBLEM AND PRACTICE QUIZ SOLUTIONS 


Solutions to the Practice quiz: Gram-Schmidt process 


1. a. The vector uy is orthogonal to uy, uz, and u3. Since uy = vj, then uy is orthogonal to v4. 


2. a. Since the vectors are already orthogonal, we need only normalize them to find 


Pars Loe a ll 1 
en {a() a4) 


1 
3. b. Let uy = vy = 1 |. Then, 
—1 
1 —2 
(ul vo)uy 2 
u2 V2 T = — 5 1 |] == 1 
uj Uy 3 
—1 —1 —1 
Normalizing the vectors, we have 
eet aL a 
1,2} = ¢< — oS 
1, U2 V3 Vb 


Solutions to the Problems for Lecture 21 


1. We bring A to reduced row echelon form: 


1110 1 1 1 0 1 
110 %1;/7>1]0 0 1 1 > 10 
1011 0 -1 0 1 0 
101 1 1 0 0 2 
0 10 -1}/—747];0 1 0 -1 
001 -1 001 -1 


The equation Ax = 0 with the pivot variables on the left-hand sides is given by 


Xy = —2xX4, X2 = Xq, 


and a general vector in the nullspace can be written as 


space is therefore given by the single vector 


x3 = X4, 


—2x4 


= X4 
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. A basis for the null 


158 APPENDIX A. PROBLEM AND PRACTICE QUIZ SOLUTIONS 
Solutions to the Problems for Lecture 22 


1. The system in matrix form is given by 


= ae | x4 = 
1, <2 2 x | =| -1 
2-4 5 8) \x, ad 


—3 6 -1 1 -—7 1, =2 0 -1 3 
1-2 2 3 -1})—>1)0 0 1 2 2. 
2 -4 5 8 —4 0 0 0 0 0 


The null space is found from the first four columns solving Au = 0, and writing the basic variables on 
the left-hand side, we have the system 


uy = 2u2+U4, U3 = —2ug; 


from which we can write the general form of the null space as 


2uz + U4 2 1 
uz 1 0 
=u + U4 
—2u4 0 —2 
U4 0 ‘I: 
A particular solution is found by solving Av = b, and we have 
01 — 202 —04 =3, 03 4+204 = —2. 


The free variables v2 and v4 can be set to zero, and the particular solution is determined to be v1 = 3 
and v3 = —2. The general solution to the underdetermined system of equations is therefore given by 


CORN 
| 
N 
| 
N 


Solutions to the Problems for Lecture 23 


1. We find 
11410 1 0 2 
A=]1 1 0 1], rref(A)=]0 1 0 -1], 
1011 0 1 -1 


and dim(Col(A)) = 3, with a basis for the column space given by the first three columns of A. 
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160 APPENDIX A. PROBLEM AND PRACTICE QUIZ SOLUTIONS 


Solutions to the Problems for Lecture 24 


1. We find the reduced row echelon from of A and A?: 


e 252) 1 @ { © a i 
gallo =P Deal A) i Tet O28 
i 2 <% a 7 Oo. @ © 2-3 
= 2 =1 <3 0 0 0 O 

oo 2 iy @ 2 

a 0 © 0 <9 

Al 1 0-1 3/>5]/0 0 1 -5 

1 =) ah Si 0 0 0 0 

a FT 4 8 O. &@ tf 


Columns one, two, and four are pivot columns of A and columns one, two, and three are pivot columns 


of AT. Therefore, the column space of A is given by 


2 3 1 
1 1 1 
Col(A) = span , Z ; 
(A) = sp i ” : 
1 —2 —1 


and the row space of A (the column space of A’) is given by 


2 —1 
3 —1 
Row(A) = span —1],] O]},]-1 
1 —1 1 
2 1 1 
The null space of A are found from the equations 
%y = —-X34+%X5, X2=X3—2X5, Xq = 2X5, 


and a vector in the null space has the general form 


—X%3 + X5 —1 1 
x3 — 2x5 1 —2 
x3 = Xx3 1] +4%5 0 
2x5 0 2 
x5 0 1 


Therefore, the null space of A is given by 


Null(A) = span 


161 
The null space of A! are found from the equations 
Xp = 2x4, XQ =2Xq, X3 = 5X4, 


and a vector in the null space has the general form 


—2x4 —2 
2x4 
= X4 
5X4 5 
X4 1 


Therefore, the left null space of A is given by 


LeftNull(A) = span 


It can be checked that the null space is the orthogonal complement of the row space and the left null 
space is the orthogonal complement of the column space. The rank(A) = 3, and A is not of full rank. 


162 APPENDIX A. PROBLEM AND PRACTICE QUIZ SOLUTIONS 
Solutions to the Practice quiz: Fundamental subspaces 


1. d. To find the null space of a matrix, bring it to reduced row echelon form. We have 


120 1 1 2 0 1 120 1 12 0 0 
241%1)-]/00 1 -1)}/7>/]/0 01 -1}/->]0 01 0 
3.6 1 1 001 -2 000 -1 000 1 
—2 
' : . 1 
With x, = —2x2, x3 = 0, and x4 = 0, a basis for the null space is ‘i 
0 


2. b. This system of linear equations is underdetermined, and the solution will be a general vector 
in the null space of the multiplying matrix plus a particular vector that satisfies the underdetermined 


system of equations. The linear system in matrix form is given by 


x1 
1201 

x2 
2 4 1 #41 — 

x3 
3 611 

x4 


12011 1 2 0 1 1 12 0 1 1 12000 
241 1%31;)/7/0 01 -1 -1]7>1]/0 01-1 -1/7>]0 0100 
3 6111 001 -2 -2 000 -1 -1 00011 
—2 
1 
A basis for the null space is 0 , and a particular solution can be found by setting the free 
0 


variable x2 = 0. Therefore, x1 = x3 = 0 and x4 = 1, and the general solution is 


xy —2 0 
x2 =4 1 ie 0 ; 
x3 0 0 
X4 0 1 


where a is a free constant. 


2 0 0 
3. c. The matrix in reduced row echelon form is | 2 4 1 OJ]. The number of 
6 0 1 


pivot columns is three, and this is the rank. 
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Solutions to the Problems for Lecture 25 


1. We first use the Gram-Schmidt process to find an orthonormal basis for W. We assign u, = 


(1 1 1)". Then, 


u= {1 


7 _ 2 
~ 3 3 
1 
An orthonormal basis for W is therefore 
1 —2 
: 1 : 1 
81 = , 82 >= 
3 6 
v3 1 1 
The projection of v onto W is then given by 
—2 
r T 1 1 
Vprojy = (V 81)81 + (v's2)82 = giatb +c) 1 + ¢(—2a +b +c) 1 
1 
When a = 1, b = c = 0, we have 
1 —2 1 
Voroi,, = : 1] - : 1]/=1]0 
PLOjyw 3 3 7 J 
1 1 0 
and when b = 1, a = c = 0, we have 
1 —2 0 
1 1 1 
Vprojw = 3 Hi + 6 1 —— 2 
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Solutions to the Problems for Lecture 26 


(i) - 


PRP PB 
WN KF OO 
Be W WR 
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Solutions to the Problems for Lecture 27 


1. The normal equations are given by 


1111 
0 12 3 


Gates 
eee: 


The solution is By = 7/5 and 6, = 9/10, and the least-squares line is given by y = 7/5 + 9x/10. 


BPR RP BR 
WN KF OO 
eww rr 


or 
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Solutions to the Practice quiz: Orthogonal projections 


1. b. We first normalize the vectors in W to obtain the orthonormal basis 


1 i ca 
w — 7 Wo = — 
ane? V6 
0 
Then the orthogonal projection of v = | 0 | onto W is given by 
1 
0 —2 —1 
T T 1 1 1 
Vprojy = (v Ww 1)W] + (v W2)W2 = _) 1 + 6 1 = 3 —1 
-1 1 2 
2. d. The overdetermined system of equations is given by 
Bot Bim = 41, 
Bo + Bix2 = y2, 
Bo + Bix3 = Y3- 


Substituting in the values for x and y, and writing in matrix form, we have 
1 1 1 
i: ¥ ( " =|1 
1 3) VPI 3 
3. b. The normal equations are given by 
111 a 111 ; 
Ge s)E |G) =O 2} 
i 3p 3 
3 6 Bo\ [5 
6 14) \p,) \12)° 


Inverting the two-by-two matrix, we have 


Multiplying out, we have 


The best fit line is therefore y = -; +x. 


Solutions to the Problems for Lecture 28 


3. Let 


Then 


and 


0 


So oO - 
oOo fF O&O 


1 


O}= Tx ixTt=1, 


dbi + ecg + fah — fbg — eai — dch 


Coo fa 
=~ SF A 
a 
lI 


= —(aei + bfg + cdh —ceg 


a-(’ a B= 
c a 


B= ae+bg af+bh 
~ \ce+dg cf+dh)’ 


a b 
bdi—afh)=—|d e fi]. 
gih 


eo: 


ae + bg)(cf + dh) — (af +bh)(ce+dg) 


acef + adeh + bcf g + bdgh) 


= (adeh + bcfg) — (adf.g + bceh) 
= ad(eh — fg) — be(eh — fg) 
= (ad — be) (eh — fg) 

= det Adet B. 


(acef + adf x + bceh + bdgh) 
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Solutions to the Problems for Lecture 29 


1. We first expand in minors across the fourth row: 


2 4 

3 2 4 #0 

9 O 4 1 0 
0 4 1 0 

8 —-5 6 7 —-2)=2 ; 

-—5 6 7 —2 

=) 2 0 0 O 
0 3 2 0 

4 0 3 2 0 


3 2: 4 0 
0 4 0 van 
2 =4/0 4 I}. 
—5 6 7 —2 
0 2 
0 3 2 0 
Finally, we expand in minors down the first column: 
4 1 
4/0 =12 = 60. 
0 3 2 
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Solutions to the Problems for Lecture 30 


1. For each element chosen from the first row, there is only a single way to choose nonzero elements 
from all subsequent rows. Considering whether the columns chosen are even or odd permutations of 
the ordered set {1,2,3,4}, we obtain 


= afhj — behj + cegj — degi. 


oo “Ss So 
r O49 
- Ooo 


oo fn 8 
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Solutions to the Problems for Lecture 31 


APPENDIX A. PROBLEM AND PRACTICE QUIZ SOLUTIONS 


1. Suppose the square matrix A has two equal rows. If we interchange these two rows, the determinant 


of A changes sign according to Property 2, even though A doesn’t change. Therefore, det A = — det A, 


or det A = 0. 


2. To prove that the determinant is a linear function of row i, interchange rows 1 and row 7 using 


Property 2. Use Property 3, then interchange rows 1 and row i again. 


3. Consider a general n-by-n matrix. Using the linear property of the jth row, and that a matrix with 


two equal rows has zero determinant, we have 


ai 


ayy + kaj 


Therefore, the determinant doesn’t change by adding k times row-i to row-j. 


Ain 


Lin + kain 


—1 2 0 
1} = |0 1 
0 -1 


+k 


7/2 


din 


Qin 


Ain 
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Solutions to the Practice quiz: Determinants 


1. a. To find the determinant of a matrix with many zero elements, perform a Laplace expansion 
across the row or down the column with the most zeros. Choose the correct sign. We have for one 


expansion choice, 


3 Se 
4. fue @ 

9.9 .2 © oO ae 2 go 2 2 

0 0-2 0 OF =-4| 2 -2 -2|=8 — 48. 

a 8 :-2e B25 is * ce o = ae 
=a: 3: +3 

i Be os 8 


2. b. We can apply the Leibniz formula by going down the first column. For each element in the first 
column there is only one possible choice of elements from the other three columns. We have 


0 


= afhj — behj + cegj — degi. 


a oO fF 2 
oom; 8 
Oo rN GO 


j 
The signs are obtained by considering whether the following permutations of the rows {1,2,3,4} are 


even or odd: afhj = {1,2,3,4} (even); behj = {2,1,3,4} (odd); cegj = {3,1,2,4} (even); degi = 
{4,1,2,3} (odd). 


3. c. The only identity which is false is det (A + B) = det A + det B. 
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Solutions to the Problems for Lecture 32 
1. The characteristic equation is given by 
0=det(A—AI)=| d e—-A f 
h i-A 


(a—A)(e—A)(i— A) + bf g + cdh — c(e — A)g — bd(i— A) — (a— A) fh 
AP + (ate+ija? — (ae + ai + ei — bd — cg — fh)A + aei + bfg + cdh — ceg — bdi —afh. 


The result can be made more memorable if we recall that 


Tr{A} =a+e+i, det A = aei + bfg +cdh — ceg — bdi—afh. 


The coefficient of the A term can be rewritten as 


ae + ai + ei — bd —cg — fh = (ei— fh) + (ai — cg) + (ae bd) =|" f 


h 


which are the sum of the minors obtained by crossing out the rows and columns of the diagonal 


elements. The cubic equation in more memorable form is therefore 


AS — Tr{ A} A? + ) "(minors of the diagonal elements of A)A — det A = 0. 
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Solutions to the Problems for Lecture 33 


2. 7 
1. Let A = (; ) . The eigenvalues of A are found from 


2-—A 7 


0 = det (A — Al) = a 


= (2—A)* — 49. 


Therefore, 2— A = +7, and the eigenvalues are A; = —5, Ay = 9. The eigenvector for A; = —5 is 


found from 


or xX; + x2 = 0. The eigenvector for Az = 9 is found from 


len 


or xX; — X2 = 0. The eigenvalues and corresponding eigenvectors are therefore given by 


A 5 : A. =9 : 
=-dJ, x1 = , =7, n= a 
i 1 4 2 2 1 


2. The eigenvalues are found from 


i ae 0 
O=det(A-AI)=|] 1 2-A 1 = (2—A) ((2—a)?-2). 
0 i 2=A 


Therefore, A, = 2, Ay =2— V2, and A3 =2+ /2. The eigenvector for A; = 2 are found from 


01 0 xy 
101 x | =0, 
01 (0 x3 
1 
or xX» = O and x; + x3 = 0, or xj = 0 |. The eigenvector for Ay = 2 — /2 is found from 
—1 
J2 1 O* fx; 
1 V2 1] |x| =0 
0 1 J/2 x3 
Gaussian elimination gives us 
v2 1 =O O 1 
rref 1 V2 1Jj= 1 v2 
0 1 2 0 oO O 
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1 
Therefore, x1 = x3 and x. = —\/2x3 and an eigenvector is x7 = | —/2]|. Similarly, the third 
1 
1 
eigenvector is x3 = | //2 


1 


Solutions to the Problems for Lecture 34 


1 
34. Let A = ( . The eigenvalues of A are found from 


—-1 1 


1-A 1 


O=det(A-AN=f 


=(1—A)?4+1. 


Therefore, 1 — A = +i, and the eigenvalues are Ay = 1 —i, Ag = 1+. 
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176 APPENDIX A. PROBLEM AND PRACTICE QUIZ SOLUTIONS 


Solutions to the Practice quiz: The eigenvalue problem 


1. b. The characteristic equation det (A — AI) = 0 for a two-by-two matrix A results in the quadratic 
equation A? —TrAA + det A = 0, which for the given matrix yields A*—3A4+1=0. Application of 


ale, = 4 
the quadratic formula results in A+ = z 2 = : + v5 


2. d. The characteristic equation is det (A — AI) = = (3—A)*+1=0. Withi = /-1, 


the solution is A+ = 3+i. 


3. b. One can either compute the eigenvalues and eigenvectors of the matrix, or test the given possible 


answers. If we test the answers, then only one is an eigenvector, and we have 


210 1 24+V2 1 
12 1) |] V2| =|24+2v2| =@24+v2)|/ v2 
012 1 J2+2 1 
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Solutions to the Problems for Lecture 35 
1. Let A; and Az be distinct eigenvalues of A, with corresponding eigenvectors x, and x2. Write 
C1X1 + c2x2 = 0. 


To prove that x; and x» are linearly independent, we need to show that c) = cp = 0. Multiply the 
above equation on the left by A and use Ax; = Ax, and Ax2 = A2x2 to obtain 


C1A4X1 > C2A2x2 = 0. 
By eliminating x, or by eliminating x2, we obtain 
(Aq — Az)c1x1 = 0, (Az — A1)ce2x2 = 0, 


from which we conclude that if Ay A Ag, then cy = cp = 0 and x and xz are linearly independent. 


2. Let A be an n-by-n matrix. We have 
dim(Col(A)) + dim(Null(A)) = n. 


Since the columns of A are linearly independent, we have dim(Col(A)) = n and dim(Null(A)) = 0. 
If the only solution to Ax = 0 is the zero vector, then det A # 0 and A is invertible. 
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Solutions to the Problems for Lecture 36 


2 1 0 
1. The eigenvalues and eigenvectors of A= | 1 2 1] are 
01 2 


1 1 1 
M=2,xm=] O|; A2=2-v2, u=|-V2|; %=24+v24=| V2 
—1 1 1 


Notice that the three eigenvectors are mutually orthogonal. This will happen when the matrix is 
symmetric. If we normalize the eigenvectors, the matrix with eigenvectors as columns will be an 


orthogonal matrix. Normalizing the orthogonal eigenvectors (so that S-! =S!), we have 


1/V2—s 1/2 1/2 
S= 0 —1/V2 1/V2 
-1/V2 1/2 1/2 


We therefore find 


2 0 0 1/V2 0 -1/V2\ /2 1 0 1/2 1/2 1/2 
0 2-vV2 0 =! 1/2 -1//2 1/2 12 1 0 —-1/V2 1/V2 
0 0 2+ 72 1/2. 1/V2 1/2 012 -1//2 1/2 1/2 
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Solutions to the Problems for Lecture 37 


i; 
pups 
=1+SAs14 a zee bate 
= (t+A4 = ae 
= Se4s-}. 


Because A is a diagonal matrix, the powers of A are also diagonal matrices with the diagonal elements 
raised to the specified power. Each diagonal element of e’ contains a power series of the form 


2 
Aj 
2! 


AP 
Bp tee 


tA4 


which is the power series for ei, 


BeatriceGloria_personal library 
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Solutions to the Problems for Lecture 38 


1. We use the result 


to find 
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Solutions to the Practice quiz: Matrix diagonalization 


1. c. Eigenvectors with distinct eigenvalues must be linearly independent. All the listed pairs of 


1 —1 
eigenvectors are linearly independent except x; = ( ) and x2 = ( : , Where x2 = —Xj. 


oul fi a ae. ale a) 
2. d. A simple calculation shows that = = I. Therefore = = 
1 0 01 1 0 1 0 


po =I. 


A more complicated calculation diagonalizes this symmetric matrix. The eigenvalues and orthonor- 


1 /1 1 1 
— and Ay = —1, v2 = — . The diagonal- 
v2 (;) eve ’ 


ization then takes the form a = u : : : : : . Then, 
1 0 2\1 -1 0 -1 1 -1 
100 100 
01 i 1 1 1 0 1 1\ 1/1 1 1 0 1 1 “1 
1:0) 240 =1/ \0 <1 ‘a 2 Sa Wo ty Va ay 


3. a. We have 


mal eigenvectors are found to be A, = 1, v7 = 


